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Costs  of  training  military  personnel  in  maintenance  skills  comprise 
a  significant  portion  of  total  military  training  costs.  The  effective¬ 
ness  of  maintenance  training  is  currently  measured  only  by  student 
achievement  at  schools,  and  correlations  between  school  achievement  and 
on-the-job  performance  have  not  been  established. 

The  military  services  currently  employ  extensive  systems  for  day-to- 
dav  management  of  maintenance  operations.  If  the  data  generated  and 
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reported  through  these  systems  could  be  used  to  estimate  the  performance 
of  either  maintenance  organizations  or  the  individuals  assigned  to  them, 
they  might  also  provide  information  needed  to  determine  the  effectiveness 
of  alternative  methods  of  training  maintenance  personnel. 

Five  maintenance  management  systems  were  investigated.  Three  of 
these  appear  to  offer  no  prospect  of  assessing  maintenance  performance, 
due  primarily  to  the  way  maintenance  organizations  are  structured.  The 
remaining  two  systems  may  provide  a  restricted  capability  for  assessing 
performance  with  some  modest  changes  to  the  data  reported  through  the 
systems.  The  effect  of  the  suggested  data-reporting  changes  would  be  to 
identify  maintenance  actions  that  are  not  suitable  for  assessing  perfor¬ 
mance,  but  the  question  remains  of  whether  the  remaining  sample  of  main¬ 
tenance  actions  would  adequately  represent  the  job  requirements  of 
different  skill  areas. 
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ABSTRACT 


Costs  of  training  military  personnel  in  maintenance  skills 
comprise  a  significant  portion  of  total  military  training  costs. 
The  effectiveness  of  maintenance  training  is  currently  measured 
only  by  student  achievement  at  schools,  and  correlations  between 
school  achievement  and  on-the-job  performance  have  not  been 
established. 

The  military  services  currently  employ  extensive  systems 
for  day-to-day  management  of  maintenance  operations.  If  the 
data  generated  and  reported  through  these  systems  could  be  used 
to  estimate  the  performance  of  either  maintenance  organizations 
or  the  individuals  assigned  to  them,  they  provide  information 
needed  to  determine  the  effectiveness  of  alternative  methods  of 
training  maintenance  personnel. 

Five  maintenance  management  systems  were  investigated. 

Three  of  these  appear  to  offer  no  prospect  of  assessing  mainte¬ 
nance  performance,  due  primarily  to  the  way  maintenance  organi¬ 
zations  are  structured.  The  remaining  two  systems  may  provide 
a  restricted  capability  for  assessing  performance  with  some 
modest  changes  to  the  data  reported  through  the  systems.  The 
effect  of  the  suggested  data-reporting  changes  would  be  to 
identify  maintenance  actions  that  are  not  suitable  for  assess¬ 
ing  performance,  but  the  question  remains  of  whether  the  remain¬ 
ing  sample  of  maintenance  actions  would  adequately  represent 
the  job  requirements  of  different  skill  areas. 
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SUMMARY 


A.  PURPOSE 


This  paper  assesses  the  possibility  of  using  data  generated 
by  the  maintenance  management  systems  of  the  military  services 
to  evaluate  the  effectiveness  of  alternative  methods  of  training 
maintenance  personnel.* 

B .  BACKGROUND 


Costs  of  training  maintenance  skills  comprise  a  signifi¬ 
cant  portion  of  the  $3  billion  spent  each  year  for  technical 
training  at  military  schools  and  can  be  expected  to  increase 
with  increases  in  the  complexity  of  weapon  systems.  On  the 
other  hand,  the  potential  costs  of  "inadequate"  maintenance,  in 
terms  of  increased  operating  costs  and  reduced  operational  capa¬ 
bilities,  may  be  considerably  greater  than  the  costs  of  provid¬ 
ing  more  extensive  and  more  effective  maintenance  training. 

Training  effectiveness  is  currently  measured  only  by  stu¬ 
dent  achievement  at  schools.  However,  the  true  effectiveness 
of  training  lies  in  the  performance  of  personnel  on-the-job,  and 
the  comparative  effectiveness  of  different  amounts  and  methods 


*In  1976,  the  Defense  Science  Board  recommended  cost-effective¬ 
ness  evaluations  of  military  training.  This  study  is  one  of 
several  undertaken  in  response  to  that  recommendation.  The 
study  was  performed  for  the  Office  of  the  Deputy  Under  Secre- 
trary  of  Defense  for  Research  and  Engineering  (Research  and 
Advanced  Technology) ,  under  the  technical  cognizance  of  the 
Military  Assistant  for  Training  and  Personnel  Technology. 
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of  training  should  be  measured  by  comparing  on-the-job  perform¬ 
ances  of  personnel  trained  in  different  ways. 

Correlations  between  school  achievement  and  on-the-job 
performance  of  maintenance  personnel  have  not  been  established, 
and  the  development  and  operation  of  a  data  system  for  this  pur¬ 
pose  would  be  a  costly  undertaking.  However,  the  military  ser¬ 
vices  currently  employ  extensive  systems  for  the  day-to-day 
management  of  their  maintenance  operations,  and  these  systems 
generate  extensive  historical  data.  If  these  data  could  be 
used  to  shed  light  on  the  performance  of  either  maintenance  or¬ 
ganizations  or  the  individuals  assigned  to  them,  they  might  also 
provide  information  that  would  be  helpful  in  determining  the 
effectiveness  of  alternative  methods  of  maintenance  training. 

C.  DATA  REQUIREMENTS 

Assessment  of  the  effectiveness  of  alternative  methods  of 
training  must  be  based  on  comparative  analysis;  this  levies 
stringent  constraints  on  the  content  and  format  of  the  histori¬ 
cal  data.  Comparisons  of  maintenance  performance  require  that; 

1.  Criteria  of  performance  can  be  formulated  and  their 
measures  quantified  from  the  historical  data; 

2.  Similar  maintenance  operations  can  be  identified  and 
grouped ;  and 

3.  Maintenance  operations  can  be  associated  with  the 
individuals  or  groups  performing  them. 

Translating  maintenance  performance  into  comparisons  of  train¬ 
ing  effectiveness  requires  that; 

1.  The  groups  of  similar  maintenance  tasks  can  be  related 
to  skill-areas  associated  with  particular  training 
programs  and 

2.  The  individuals  or  groups  performing  the  maintenance 
can  also  be  related  to  skill-areas  associated  with 
particular  training  programs. 
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These  constraints  appear  to  be  satisfied  only  when  the 
data  encompass  all  (or  a  representative  sample)  of  the  mainte¬ 
nance  operations  performed  on  all  (or  a  representative  sample) 
of  the  equipment  holding  of  a  military  service  and  provide 
unambiguous  (i.e.,  coded  or  keyword  format)  answers  to  the  fol¬ 
lowing  four  questions: 

1.  What  equipment  was  maintained?  The  equipment  must  be 
identified  at  a  sufficiently  low  level  (e.g.,  subsystem 
or  component)  so  that  maintenance  can  be  associated 
with  a  single  skill-area  that  can  be  related  to  specific 
training . 

2.  What  maintenance  operation  was  performed?  The  data 
must  separately  document  discrete  small-scale  mainten¬ 
ance  tasks  (e.g.,  remove,  disassemble)  that  are  compar¬ 
able  whenever  they  are  performed  on  the  same  subsystem 
or  assembly  installed  on  the  same  model  of  equipment 
end-item,  rather  than  documenting  a  complete  mainten¬ 
ance  action  (i.e.,  the  repair  of  an  item  of  installed 
equipment  that  permits  return  of  the  end-item  to  serv¬ 
ice)  that  may  require  different  skills  and  levels  of 
effort,  depending  upon  the  precise  nature  of  the  mal- 

f unction. 

3.  Who  performed  the  maintenance?  Maintenance  organiza¬ 
tions  are  typically  manned  to  provide  the  variety  of 
skills  required  by  all  weapon  end-items  maintained. 
Documentation  of  maintenance  must  identify  the  indivi¬ 
duals  or  elements  of  the  organization  performing  main¬ 
tenance  in  a  manner  that  can  be  related  to  skill-area 
(e.g.,  work-center). 

4 .  Why  was  maintenance  required?  Reasons  for  performing 
the  maintenance  may  (or  may  not)  be  required  to  deter¬ 
mine  comparability  between  maintenance  tasks.  For 
example,  the  task  of  removing  a  particular  subsystem 

or  assembly  from  an  end-item  is  the  same,  regardless  of 


» 
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why  the  assembly  is  removed,  while  the  skill  and  effort 
required  to  effect  its  repair,  per  se ,  may  differ 
widely  as  a  function  of  why  maintenance  is  required. 


D.  THE  IMPACT  OF  TEAM  AND  CROSS-SKILL  MAINTENANCE 

Beyond  considerations  of  data  requirements,  two  common 
practices  in  military  maintenance  preclude  an  assessment  of 
training  effectiveness  --  team  maintenance  ( the  assignment  of 
two  or  more  individuals  to  a  maintenance  task)  and  cross-skill 
maintenance  (performance  of  maintenance  in  one  skill-area  by 
personnel  who  have  been  trained  in  other  skills) .  Several 
reasons  lie  behind  team  maintenance,  including  on-the-job 
training  (OJT) ,  size  of  equipment  undergoing  maintenance  or 
its  proximity  to  test  equipment,  expediency  in  returning  end- 
items  to  operational  status,  and  the  utilization  of  maintenance 
facilities.  In  assessing  training  effectiveness,  interest  must 
center  on  recently  trained  individuals  before  the  differential 
effectiveness  of  different  methods  and  amounts  of  training  is 
diluted  by  subsequent  OJT  and  experience.  However,  maintenance 
teams  would  typically  be  composed  of  individuals  with  differing 
levels  of  experience  and/or  training  and  the  recently  trained 
personnel  would  have  little  impact  on  overall  team  performance. 

In  addition,  there  is  no  systematic  way  to  quantify  the  separate 
contributions  of  team  members. 

A  number  of  reasons  lie  behind  cross-skill  maintenance;  a 
primary  one  is  that  of  providing  personnel  with  OJT  in  addi¬ 
tional  skill-areas  to  enhance  maintenance  organizations'  capa¬ 
bilities  to  satisfy  the  increased  (and  more  variable)  work¬ 
loads  anticipated  under  combat  conditions.  However,  the  effec¬ 
tiveness  of  training  in  a  given  skill-area  can  be  assessed  only 
where  maintenance  performance  is  measured  in  tasks  in  that  skill- 
area  by  individuals  who  have  been  trained  in  that  skill. 


Since  neither  team  nor  cross-skill  maintenance  provide  a 
valid  basis  for  assessing  training  effectiveness,  historical 
maintenance  data  would  have  to  allow  for  identification  of 
occurrences  of  both  so  that  their  occurrences  could  be  elimi¬ 
nated  from  analyses. 

E.  EVALUATION  OF  MAINTENANCE  MANAGEMENT  SYSTEMS 

Five  maintenance  management  systems  were  investigated: 

The  Army  Maintenance  Management  System  (TAMMS) ,  the  Navy  Ships’ 
Material  Maintenance  Management  (Ships'  3-M)  system,  the  Navy 
Aviation  Material  Maintenance  Management  (Aviation  3-M)  system, 
and  the  Air  Force  66-1  and  66-5  systems.  Taken  together,  these 
systems  control  the  management  of  organizational  and  intermedi¬ 
ate  maintenance  of  all  military  aircraft,  all  Army  and  Air  Force 
ground  equipment  ( including  missiles) ,  and  all  Navy  ships  and 
shipboard  equipment  (except  nuclear  missiles).  Table  S~ 1  dis¬ 
plays  the  attributes  of  the  data  each  generates  in  terms  of  the 
data  requirements  discussed  above.  (Since  the  Air  Force  66-1 
and  66-5  systems  employ  the  same  reporting  formats  and  codes 
they  are  combined  in  the  table.) 

On  the  basis  of  data  reported,  the  Ships'  3-M  system  ap¬ 
pears  to  have  no  capability  for  assessing  maintenance  perform¬ 
ance  (and,  hence  training  effectiveness)  for  reasons  that  encom¬ 
pass  both  the  nature  and  extent  of  data  reported.  A  critical 
point  is  that  the  majority  of  maintenance  activities  are  not 
reported  in  this  system. 

TAMMS  also  appears  to  provide  no  capability  for  assessing 
training  effectiveness.  Army  maintenance  employs  extensive 
cross-skill  and  team  maintenance.  Since  the  reported  data  con¬ 
tain  no  means  for  identifying  the  skill-areas  (and  hence  training) 
of  personnel  performing  particular  maintenance  tasks,  it  appears 
impossible  to  track  cases  of  cross-skill  work.  Further,  main¬ 
tenance  organizations  are  staffed  to  rely  on  team  maintenance 
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involving  different  maintenance  echelons  under  combat  conditions, 
and  it  is  unclear  how  its  occurrence  could  be  traced  in  the  his¬ 
torical  records. 

The  Aviation  3-M  and  Air  Force  systems  satisfy  data  re¬ 
quirements  set  out  in  Table  S-l  and  may  provide  some  capabili¬ 
ties  for  assessing  training  effectiveness.  The  principal  ques¬ 
tion  involves  whether  team  and  cross-skill  maintenance  can  be 
identified,  and  this  should  be  facilitated  with  some  seemingly 
modest  changes  to  the  data  reported  and  by  supplementing  those 
data  with  information  that  is  normally  available  from  unit  roster 
and  personnel  record  systems. 

The  Air  Force  66-1  and  66-5  systems  currently  document 
maintenance  "crew  size"  and  start  and  stop  times  and  provide  a 
separate  entry  for  all  crew  changes  and  work  interruptions  that 
occur  during  the  course  of  a  maintenance  task.  These  data,  if 
accurately  recorded,  should  provide  a  satisfactory  separation 
of  team  maintenance  from  individually  performed  maintenance. 
Aviation  3-M  currently  provides  no  permanently  retained  informa¬ 
tion  regarding  the  number  of  individuals  involved.  However, 
the  similarities  in  equipments,  their  maintenance  requirements, 
and  maintenance  organizations  between  USAF  and  Naval  aircraft 
suggest  that  information  on  the  composition  of  maintenance 
crews  could  be  readily  incorporated  into  Aviation  3-M  data. 

Identification  of  cross-skill  maintenance  presents  a  more 
formidible  problem.  Naval  aircraft  and  Air  Force  maintenance 
organizations  are  structured  internally  according  to  skill  so 
that  occurrences  of  cross-skill  work  should  be  identifiable. 
However,  a  sufficient  number  of  exceptions  (conditions  under 
which  individuals  would  be  assigned  to  tasks  outside  their 
skill-areas)  were  noted  in  documentation  of  these  systems  to 
weaken  the  case  for  identifying  cross-skill  work.  Cross-skill 
maintenance  is  a  formidable  problem  for  Air  Force  units  operat¬ 
ing  within  the  66-5  system  where  development  of  cross-skill 
capabilities  is  a  policy  objective. 


S-7 


This  would  present  no  problem  if  the  individuals  performing 
maintenance  could  be  positively  identified  (through,  e.g.,  name 
or  social  security  number) .  At  present,  neither  Aviation  3-M 
nor  the  Air  Force  systems  provide  this  identification,  and 
provisions  of  the  Privacy  Act  prevent  naming  of  individuals 
in  permanent  central  historical  data  files.  However,  it  does 
not  prevent  such  identification  in  local  unit  temporary  records 
(such  as  work-in-process  logs).  Local  unit  naming  of  personnel 
should  provide  an  early  positive  screening  for  instances  of 
cross-skill  maintenance.  For  example,  individuals  could  be 
preselected  for  analysis  of  their  performance  on  the  basis  of 
training  and  experience  information  gleaned  from  personnel 
records.  Where  maintenance  documentation  identified  these  indi¬ 
viduals  and  where  neither  team  nor  cross-skill  maintenance  was 
indicated,  the  local  records  could  be  duplicated  for  analysis 
of  performance  and  training  effectiveness,  with  the  personnel 
identification  being  deleted.  The  Navy  is  currently  modifying 
the  local  unit  portion  of  Aviation  3-M  in  a  manner  that  appears 
compatable  with  this  concept,  and  there  is  no  reason  to  believe 
that  the  Air  Force  management  systems  are  basically  incompatable 
with  it. 

Even  with  these  changes,  the  nagging  question  remains  of 
whether  the  resulting  performance  samples  would  be  representative 
of  the  job  requirements  of  the  various  skill  areas.  The  prospects 
are  not  promising,  especially  for  organizational  echelon  main¬ 
tenance.  Recently  trained  personnel  typically  are  not  assigned 
to  independent  work  on  aircraft  equipment,  and  this  is  the  pop¬ 
ulation  of  interest  for  assessing  training  effectiveness.  Team 
maintenance  is  prompted  by  features  of  the  equipment  and  enhances 
unit  effectiveness.  Cross-skill  maintenance  contributes  to  a 
maintenance  organization’s  potential  under  combat  conditions. 
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I .  INTRODUCTION 


Training  costs  devoted  to  maintenance  comprise  a  signifi¬ 
cant  portion  of  the  $3  billion  spent  each  year  for  technical 
training  at  military  schools  and  can  be  expected  to  increase 
with  increases  in  the  complexity  of  weapon  systems.  On  the 
other  hand,  the  potential  costs  of  "inadequate"  maintenance, 
in  terms  of  increased  operating  costs  and  reduced  operational 
capabilities,  may  be  considerably  greater  than  the  costs  of 
providing  more  extensive  and  more  effective  maintenance  training 

Student  achievement  in  courses  at  training  schools  is  cur¬ 
rently  used  as  the  measure  of  training  effectiveness.  However, 
the  effectiveness  of  maintenance  training  is  ultimately  deter¬ 
mined  by  the  performance  of  maintenance  personnel  in  the  field 
rather  than  at  schools.  It  follows,  then,  that  the  comparative 
effectiveness  of  different  amounts  and  methods  of  maintenance 
training  should  be  measured  by  assessing  the  comparative  perform¬ 
ance  of  maintenance  personnel  in  field  assignments  rather  than 
by  assessing  their  achievement  in  school.  Of  particular  inter¬ 
est  here  would  be  whether  the  use  of  maintenance  simulators  at 
schools  improves  the  maintenance  performance  of  students  in  the 
field  over  those  trained  with  actual  equipment. 

Correlations  between  school  achievement  and  job  perform¬ 
ance  have  not  been  established;  no  reports  exist  that  relate 
performance  on-the-job  to  training  in  a  systematic  manner.  The 
development  and  operation  of  a  data  system  for  this  purpose 
would  be  a  costly  undertaking,  and  for  that  reason  it  is  rele¬ 
vant  to  ask  whether  maintenance  data  collected  for  other  pur¬ 
poses  might  also  provide  data  on  the  effectiveness  of  mainte¬ 
nance  training.  The  services  currently  employ  large  maintenance 


management  systems  that  vary  with  respect  to  formats,  methods, 
and  completeness  of  data  recording  and  reflect  differences  in 
types  of  equipments  held  and  the  environments  in  which  they  are 
operated  and  maintained.  All  Army  equipment  is  managed  through 
the  Army  Maintenance  Management  System  (TAMMS).  The  Navy 
operates  two  Maintenance  Material  Management  (3-M)  systems — 
one  for  aircraft  and  one  for  ships. 

The  Air  Force  uses  two  maintenance  management  systems — the 
"66-1"  and  "66-5".  (66-1  and  66-5  refer  to  the  numbers  of  the 

Air  Force  Regulations  and  Manuals  that  set  out  policy  for  each 
system.)  However,  both  systems  employ  the  same  set  of  mainte¬ 
nance-reporting  formats  and  codes  and  the  same  data-processing 
programs.  As  a  result,  they  should  generate  equivalent  mainte¬ 
nance  data  and,  through  most  of  this  discussion,  are  treated  as 
a  single  entity,  denoted  as  66-1/66-5. 

All  of  these  systems  are  concerned  primarily  with  the 
daily  management  of  maintenance  organizations  and  with  attain¬ 
ing  a  high  level  of  operational  readiness  throughout  the  force. 
Operational  readiness  encompasses  considerably  more  than  mainte¬ 
nance,  per  se,  and  these  systems  address  a  variety  of  functions 
associated  with  the  general  area  of  logistics,  including: 

•  Manning  and  manpower  utilization 

•  Weapon  system  operation  or  utilization  rates 

•  Weapon  system  logbook  and  inventory  control 

•  Modification  and  configuration  control 

•  Preventive  maintenance 

•  Reliability  and  safety 

•  Technical  documentation 

•  Supply  support  and  material  control. 

It  should  be  noted  that  none  of  these  systems  was  intended  (or 
design'  d)  to  reflect  training  effectiveness.  To  the  extent 
that  they  may  provide  this  capability  is  serendipity. 

Both  the  Navy  Aviation  3-M  and  the  Air  Force  66-1/66-5  sys¬ 
tems  compile  permanent  historical  records  of  all  maintenance 
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performed.  In  the  Ships'  3-M  system  an  unknown  proportion  of 
maintenance  is  compiled  in  historical  files.  In  TAMMS,  informa¬ 
tion  is  recorded  at  the  local  level  on  all  maintenance  activity. 
However,  these  records  are  retained  only  for  a  limited  time, 
and  centralized  reporting  is  limited  to  modifications  of  equip¬ 
ment.  If  these  historical  records  can  be  used  to  assess  main¬ 
tenance  performance  (of  either  individuals  or  organizations), 
they  may  also  provide  information  that  can  be  associated  with 
the  effectiveness  of  alternative  methods  of  maintenance  training. 
Interest  must  center  on  the  performance  of  recently  trained  per¬ 
sonnel,  since  subsequent  on-the-job  training  and  field  experience 
will  dilute  the  impact  of  formal  training  on  performance.  As 
a  result,  it  is  important  to  know  when  personnel  complete  train¬ 
ing  school  courses. 

Assessing  maintenance  performance  requires:  (1)  that 
criteria  of  maintenance  performance  can  be  formulated  and  (2) 
that  maintenance  data  are  formatted  in  historical  records  in 
ways  that  can  be  related  to  the  criteria.  Training  effective¬ 
ness  might  then  be  assessed  by  comparative  analyses  of  mainte¬ 
nance  performance  between  sets  of  individuals,  that  are  differ¬ 
entiated  on  the  basis  of  training  received,  on  the  same  or 
similar  sets  of  maintenance  operations,  provided  that  (1)  data 
are  formatted  in  historical  records  so  that  like  or  similar 
maintenance  jobs  can  be  identified  and  grouped,  (2)  the  groups 
of  similar  jobs  can  be  related  to  skill  areas  associated  with 
particular  training  programs,  (3)  the  groups  associated  with 
each  skill  area  provide  a  representative  sample  of  the  mainte¬ 
nance  operations  performed  within  that  skill  area,  (4)  the 
historical  records  associate  maintenance  jobs  performed  with 
individuals  or  organizations  performing  them,  and  ( 5)  the 
individuals  or  organizations  can  be  linked  to  skill  areas 
associated  with  training  programs. 

The  approach  adopted  here  is  to  formulate  criteria  of  per¬ 
formance  of  maintenance  personnel  and  to  identify  the  types  of 
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data  that  would  permit  relating  observed  on-the-job  maintenance 
performance  to  type  of  training.  The  four  maintenance  data  re¬ 
porting  systems  are  then  examined  to  see  how  closely  they  pro¬ 
vide  the  required  data.  Only  organizational  and  intermediate 
maintenance  echelon  reporting  systems  are  considered,  since 
depot  maintenance  is  accomplished,  typically,  by  civilian  per¬ 
sonnel.  Throughout  the  discussion  maintenance  action  is  defined 
as  "all  effort  associated  with  the  completion  of  maintenance 
requirement,"  e.g.,  the  correction  of  a  malfunction  permitting 
return  of  equipment  to  an  operational  status.  Maintenance 
task  is  defined  as  "a  single  procedure  that  is  performed  as 
part  of  a  maintenance  action,"  e.g.,  remove,  troubleshoot, 
repair,  and  install  are  discrete  parts  of  a  single  maintenance 
action  that  corrects  an  equipment  malfunction. 


II.  CRITERIA  OF  PERFORMANCE 


Many  criteria  could  possibly  serve  as  partial  or  general 
indicators  of  maintenance  performance.  Only  three  are  discussed 
below,  on  the  basis  that  they  are  familiar  and  quantifiable, 
have  general  applicability,  and  offer  widely  varying  capabili¬ 
ties  to  isolate  maintenance  performance  from  other  factors  that 
represent  force  operating  capabilities. 

A.  OPERATIONAL  (COMBAT)  READINESS 

This  is  the  ultimate  (and  commonly  used)  criterion  of  over¬ 
all  unit  performance.  Readiness  is  a  global  measure  that  is 
affected  by  a  wide  range  of  influences  other  than  maintenance, 
such  as  operating  budgets,  availability  of  spare  parts,  equip¬ 
ment  operator  skills,  and  system  reliability.  The  amount  of 
time  equipment  is  not  operationally  ready  due  to  maintenance 
requirements  (NORM)  is  a  commonly  used  measure  of  unit  mainte¬ 
nance  performance;  however,  it  appears  impossible  to  relate  the 
time  periods  an  equipment  is  in  a  notoperationally-ready  or  a 
reduced-material-condition  status  to  particular  maintenance 
jobs  or  to  equipment  subsystems  or  maintenance  skill  areas. 

B.  MAINTENANCE  MAN-HOURS  PER  OPERATING  HOUR 

This  measure  is  typically  associated  with  evaluations  of 
weapon  reliability  and  maintainability.  It  can  provide  a  meas¬ 
ure  of  maintenance  performance  when  man-hours  expended  maintain¬ 
ing  particular  subsystems  can  be  compared  among  units  operating 
similar  equipment.  However,  it  provides  only  a  gross  measure 
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of  performance  since  it  cannot  distinguish  the  impacts  of  differ¬ 
ent  operating  environments  on  maintenance  requirements;  operat¬ 
ing  environments  would  have  to  be  normalized  in  order  to  assess 
effectiveness  of  different  methods  of  maintenance  training. 

C.  MAINTENANCE  MAN-HOURS  PER  MAINTENANCE  REQUIREMENT  (ACTION 
OR  TASK) 

This  measure  appears  superior  to  the  other  two  in  isolating 
maintenance  performance  in  a  way  that  might  permit  performance 
to  be  related  to  training  effectiveness,  especially  where  re¬ 
quirements  can  be  stated  in  terms  of  maintenance  tasks  (as  de¬ 
fined  above).  Each  of  the  four  reporting  systems  examined  pro¬ 
vides  explicit  statements  of  man-hours  expended  in  each  recorded 
maintenance  requirement.  This  measure  is  adopted  as  the  sole 
criterion  of  human  performance  in  the  following  discussion. 
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III.  DATA  REQUIREMENTS  FOR  ASSESSING  MAINTENANCE  PERFORMANCE 


Assessment  of  training  effectiveness  must  be  based  on  com¬ 
parative  analysis.  The  validity  of  the  analysis  will  rest  upon 
limiting  comparisons  to  performances  of  maintenance  personnel 
on  similar  maintenance  operations  (i.e.,  similar  processes 
applied  to  the  same  models  of  equipment)  where  personnel  have 
been  trained  in  the  same  skill  areas,  but  perhaps  by  different 
methods  or  to  different  extents.  Verifying  that  these  condi¬ 
tions  are  met  places  several  constraints  on  the  data  that  should 
be  developed,  as  follows. 

A.  CODING  OF  DATA 

The  data  must  provide  unambiguous  answers  to  four  questions 
regarding  each  maintenance  operation: 

•  What  equipment  was  maintained? 

•  What  was  done  to  it  (i.e.,  the  nature  of  the  maintenance 
performed) ? 

•  Why  was  maintenance  required  (i.e.,  the  nature  of  the 
equipment  malfunction) ? 

•  Who  performed  the  maintenance? 

Unambiguous  answers  imply  that  data  be  reported  through  keyword 
or  code  systems.  Note,  though,  that  use  of  codes  and  keywords 
places  limits  on  the  way  and  the  detail  with  which  maintenance 
may  be  described  and  these  limits  may  introduce  other  ambigui¬ 
ties  into  the  data.  This  consideration  forms  the  basis  for 
some  of  the  remaining  data  constraints. 


B.  IDENTIFICATION  OF  EQUIPMENT 


The  data  must  identify  the  equipment  maintained  at  a  suf¬ 
ficiently  low  level  (e.g.,  subsystem  or  assembly)  so  that  it 
can  be  associated  with  a  single  skill  area  relatable  to  specific 
training  programs.  Maintenance  of  complex  military  end-items 
involves  a  wide  range  of  maintenance  skills,  so  that  identifica¬ 
tion  of  end-items  provides  no  information  regarding  the  skill 
area  employed.  However,  end-items  are  typically  composed  of 
a  number  of  subsystems  and  assemblies  that  can  be  associated 
with  single  skill  areas,  and  individual  maintenance  tasks  nor¬ 
mally  involve  a  single  subsystem  or  assembly.  As  a  result, 
maintenance  documentation  may  shed  light  on  training  effective¬ 
ness  only  if  the  equipment  undergoing  maintenance  is  identified 
in  a  way  that  also  identifies  the  associated  skill  area. 

A  further  point  with  regard  to  equipment  identification 
is  that  comparisons  of  performance  are  valid  only  where  the 
same  item  of  equipment  is  involved.  However,  assuring  that 
equipment  is  the  same  may  require  more  than  the  use  of  a  code 
structure.  Regardless  of  its  complexity,  a  code  system  may  be 
insensitive  to  differences  in  end-item  or  subsystem  configura¬ 
tions  (such  as  the  use  of  different  models  of  "black  boxes" 
with  similar  physical  and  operating  characteristics  or  different 
locations  of  the  same  model  of  black  box)  that  can  require  dif¬ 
ferent  maintenance  procedures.  Where  equipment  is  not  standard¬ 
ized,  information  from  outside  the  reporting  system  may  be  re¬ 
quired  to  assure  that  configurations  are  identical. 

C.  DESCRIPTION  OF  MAINTENANCE  PERFORMED 

Describing  what  was  done  in  the  course  of  maintenance  in¬ 
volves  two  considerations.  The  first  is  simply  that  sufficient 
code  values  are  available  to  adequately  distinguish  between 
different  maintenance  requirements.  The  second  concerns  the 


magnitude  of  maintenance  effort  that  is  separately  documented. 
Individual  maintenance  records  can,  alternatively,  document 
each  of  the  small-scale  discrete  maintenance  tasks  comprising 
a  maintenance  action,  some  partial  aggregation  of  tasks  required 
by  a  maintenance  action,  or  a  complete  maintenance  action.  How¬ 
ever,  to  evaluate  maintenance  performance,  the  records  must 
separately  document  discrete  and  well-defined  maintenance  opera¬ 
tions  that  are  comparable  whenever  they  are  performed  on  the 
same  subsystem  or  assembly  (black  box)  installed  on  the  same 
model  of  equipment  end-item. 

Typically,  military  equipment  is  complex,  and  a  wide  range 
of  maintenance  requirements  can  be  associated  even  with  rela¬ 
tively  small  subsystems  and  assemblies.  The  use  of  code  systems 
may  place  severe  limitations  on  the  ways  available  to  describe 
what  is  entailed  in  maintenance  actions.  Different  actions, 
that  can  best  (or  only)  be  described  by  the  same  code  value, 
may  require  significantly  different  levels  of  skill  and  amounts 
of  effort.  For  example,  low  power  output  from  a  particular 
electronic  assembly  may  arise  from  a  number  of  causes,  and  the 
skills  and  effort  required  to  accomplish  correction  will  vary 
with  the  cause.  In  some  cases,  correction  may  be  accomplished 
without  removing  equipment  from  the  end-items  on  which  they  are 
installed,  i.e.,  on-equipment  repair.  In  other  cases,  it  may 
be  necessary  to  remove,  "bench"  repair,  and  reinstall,  i.e., 
make  off-equipment  repair.  While  these  are  not  comparable 
maintenance  efforts,  the  differences  between  them  are  hidden  by 
documenting  only  the  overall  task,  e.g.,  repair  of  the  assembly. 

Maintenance  actions  involve  discrete  smaller  scale  mainte¬ 
nance  tasks  (e.g.,  remove,  disassemble,  troubleshoot,  replace 
parts,  assemble,  reinstall)  that  are  well  defined,  recur  fre¬ 
quently  as  elements  of  disparate  maintenance  actions,  and  con¬ 
stitute  comparable  maintenance  when  applied  to  one  model  of 
subsystem  or  assembly  installed  on  one  model  of  end-item.  As 
a  practical  matter,  documentation  of  standardized  tasks  appears 
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to  be  a  necessary  condition  for  assessing  maintenance  perform¬ 
ance.  Further,  the  number  of  different  standardized  tasks 
involved  in  performing  maintenance  appears  small  enough  to  be 
incorporated  within  feasible  code  systems. 

D.  DESCRIPTION  OF  MALFUNCTIONS 

Reasons  for  performing  maintenance  (i.e.,  the  nature  of 
malfunctions)  may  or  may  not  be  necessary  to  identify  comparable 
maintenance  tasks,  depending  upon  the  type  of  task  performed. 

For  example,  the  task  of  removing  a  particular  subsystem  or 
assembly  (e.g.,  a  communications  receiver)  from  a  given  model 
of  aircraft  involves  the  same  skill  and  the  same  amount  of  work, 
regardless  of  the  reason  maintenance  is  required.  On  the  other 
hand,  the  skills  and  effort  involved  in  the  actual  repair  of 
the  assembly  may  differ  widely  as  a  function  of  why  the  mainte¬ 
nance  is  required  (e.g.,  consider  the  difference  between  replac¬ 
ing  a  burned  out  indicator  bulb  and  correcting  RF  signal  distor¬ 
tions)  . 

We  have  no  way  of  quantifying  the  proportion  of  maintenance 
tasks  for  which  malfunction  descriptions  will  be  necessary.  The 
proportion  probably  varies  with  the  type  of  system  being  main¬ 
tained.  However,  it  can  be  expected  that  such  descriptions 
will  be  required  where  maintenance  involves  repair,  per  se,  of 
equipment . 

E.  IDENTIFICATION  OF  PERSONNEL 

The  data  must  identify  organizations  (or  individuals)  per¬ 
forming  maintenance  in  a  way  that  will  allow  their  association 
with  skill  areas  that  can  be  related  to  specific  training  pro¬ 
grams.  Documentation  of  individual  maintenance  tasks  can  alter¬ 
natively  identify  the  individual  performing  maintenance  (who 
can  be  associated  with  skill  area  and  training  received  through 
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other  data,  such  as  personnel  records),  work  centers  or  shops 
that  are  manned  by  personnel  predominantly  trained  in  a  single 
skill  area,  or  work  centers  and  maintenance  organizations  that 
encompass  a  range  of  skill  areas.  Cases  of  maintenance  perform¬ 
ance  that  are  valid  for  assessing  training  effectiveness  are 
limited  to  those  in  which  the  skill  area  required  by  the  task 
(as  identified  through  equipment  codes)  is  the  same  as  that  in 
which  the  personnel  performing  the  task  have  been  trained.  As 
a  result,  maintenance  documentation  can  shed  light  on  training 
effectiveness  only  if  personnel  performing  maintenance  can  be 
identified  in  a  way  that  also  identifies  their  skill  areas. 

1.  Single  Individuals 

If  individuals  were  identified  in  maintenance  documentation, 
assessment  of  training  effectiveness  could  be  based  on  direct 
comparisons  of  the  performance  of  personnel,  trained  in  the 
same  skill  area  by  different  methods  or  to  different  extents, 
on  similar  maintenance  tasks.  However,  the  prospects  for  mak¬ 
ing  assessments  in  this  manner  appear  small,  provisions  of 
the  Privacy  Act  prevent  reporting  of  the  performance  of  individ¬ 
uals  to  service-wide  data  bases  (such  as  the  central  files  of 
the  maintenance  management  systems) .  If  local  unit  records 
identify  individuals,  the  records  cannot  be  collated  and  evalu¬ 
ated  above  some  intermediate  organizational  level.  While  this 
does  not  prevent  their  use  for  evaluation  of  training  effective¬ 
ness,  it  implies  recourse  to  special  data  collection  or  process¬ 
ing  methods. 

2.  Work  Center 

Where  individuals  are  not  identified  in  maintenance  docu¬ 
mentation,  theoretically  it  should  be  possible  to  provide  some 
measure  of  training  effectiveness,  if  the  documentation  identi¬ 
fied  the  work  center  (WC)  performing  maintenance,  for  those 
WCs  that  are  predominantly  manned  by  personnel  trained  in  a 
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single  skill  area.  Different  maintenance  organizations  reporting 
through  the  same  management  system  have  similar  WC  structures. 

If  (1)  WCs  could  be  characterized  in  terms  of  the  training  re¬ 
ceived  and  experience  possessed  by  their  assigned  personnel,  and 

(2)  if  samples  were  gathered,  across  different  maintenance 
organizations,  of  the  performance  of  the  same  WC  performing 
the  same  maintenance  task  on  the  same  item  of  equipment,  then 

(3)  inferences  regarding  training  effectiveness  might  be  drawn 
by  regressing  WC  performance  against  training  and  experience 
profiles  within  the  context  of  models  of  the  impact  of  training 
on  performance. 

The  likelihood  that  differences  in  performance  among  WCs 
can  be  confidently  attributed  to  differences  in  training  does 
not  appear  promising.  The  impact  of  differences  in  training 
programs  are  most  relevant  for  recently  trained  personnel,  and 
such  personnel  will  have  relatively  little  impact  on  overall 
WC  performance.  A  number  of  other  factors,  internal  to  the 
maintenance  organization,  affect  maintenance  performance  (e.g., 
the  quality  of  supervision  and  the  extent  to  which  daily  work¬ 
loads  are  used  for  OJT) .  It  could  well  be  that  sample  sizes 
necessary  to  attribute  significance  to  the  impact  of  training 
could  not  be  obtained. 

3.  Maintenance  Organization 

Where  maintenance  documentation  identifies  only  maintenance 
organizations  (or  WCs  that  are  associated  with  a  number  of  skill 
areas)  it  would  appear  impossible,  as  a  practical  matter,  to 
relate  performance  to  training  effectiveness.  All  the  consider¬ 
ations  discussed  in  regard  to  skill-related  WCs  also  are  rele¬ 
vant  here.  In  addition,  weapon  systems  are  complex  and  require 
the  application  of  a  number  of  maintenance  skills;  maintenance 
organizations  are  generally  large  enough  to  encompass  a  large 
number  of  skill  areas,  and  the  proportion  of  total  personnel 
trained  in  a  single  skill  area  would  typically  be  small.  While 
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theoretically  it  may  be  possible  to  assess  training  effective¬ 
ness  through  modelling  and  regression  analysis,  it  appears  that 
requisite  sample  sizes  would  be  much  larger  than  those  required 
when  skill-related  WCs  are  identified. 

F.  EXTENT  OF  MAINTENANCE  DOCUMENTATION  AND  REPORTING 

The  job  requirements  of  any  maintenance  specialty  encompass 
a  host  of  distinct  tasks  employing  a  variety  of  skills  which  a 
training  program  must  provide.  Assessment  of  training  effec¬ 
tiveness  must  address  performance  across  a  sufficiently  wide 
range  of  tasks  to  provide  a  representative  sample  of  the  skills 
that  are,  in  fact,  employed  on  the  job.  While  documentation  of 
all  maintenance  would  provide  such  a  representative  sample,  it 
is  not  necessarily  a  requirement  for  daily  management  and  may 
interfere  with  higher  priority  functions  of  a  maintenance  unit. 
Efficient  unit  operations  may  imply  a  systematic  exclusion  of 
documentation  of  certain  types  of  maintenance,  documentation 
for  certain  types  of  equipment  (end-items  or  subsystems),  and 
documentation  of  equipments  in  certain  types  of  assignments. 

In  these  cases,  data  that  are  gathered  may  not  encompass  all 
relevant  skills,  and  the  extent  to  which  they  fall  short  may 
not  be  discernible.  In  addition,  local  level  documentation 
may  not  be  fully  reported  to  higher  echelons  or  centralized 
data  banks.  Such  an  unavailability  of  data  at  more  convenient 
sources  reduces  the  practical  usefulness  of  the  maintenance 
management  systems  for  assessing  training  effectiveness. 

G.  IDENTIFICATION  OF  TEAM  AND  CROSS-SKILL  MAINTENANCE 

Team  maintenance  (the  assignment  of  two  or  more  individuals 
to  a  maintenance  task)  and  cross-skill  maintenance  (the  assign¬ 
ment  of  personnel  trained  in  one  skill  area  to  a  maintenance 
task  that  is  associated  with  a  different  skill  area)  appear  to 
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be  common  practices,  but  both  are  intractable  constraints  for 
assessing  training  effectiveness.  A  number  of  reasons  lie  be¬ 
hind  team  maintenance.  It  provides  a  vehicle  for  OJT  within 
the  normal  flow  of  shop  work;  some  equipments  are  too  large  to 
be  handled  by  one  person;  some  maintenance  tasks  may  require 
the  simultaneous  observation  or  operation  of  equipments  at  dif¬ 
ferent  locations  (e.g.,  a  display  panel  in  the  cockpit  of  an 
aircraft  and  a  test  set  located  outside  the  aircraft) ;  it  per¬ 
mits  higher  utilization  of  maintenance  facilities  (in  particular, 
two-  or  three-shift  shop  operations).  Team  maintenance  compli¬ 
cates  assessing  training  effectiveness  for  two  reasons.  First, 
there  would  be  no  systematic  way  to  quantify  the  individual 
contributions  of  team  members  to  the  completion  of  a  single 
maintenance  task.  Second,  formal  school  training  represents 
only  the  initial  stages  in  the  development  of  an  individual's 
job  skills  and  is  generally  followed  by  OJT  at  the  individual's 
duty  station.  As  a  result,  the  effectiveness  of  different 
methods  and  amounts  of  school  training  would  be  diluted  by  sub¬ 
sequent  OJT  and  experience.  Moreover,  maintenance  teams  would 
typically  be  composed  of  individuals  with  differing  levels  of 
experience  and/or  training,  and  the  newly  trained  personnel 
would  have  little  impact  on  overall  team  performance. 

The  effectiveness  of  training  in  a  given  skill  area  can  be 
assessed  only  where  maintenance  performance  is  measured  in  tasks 
in  that  skill  area  by  individuals  who  have  been  trained  in  that 
skill.  However,  personnel  may  be  assigned  to  maintenance  tasks 
in  areas  other  than  those  for  which  they  have  been  trained  for 
two  reasons.  Daily  workloads  vary  among  the  different  WCs  of 
a  maintenance  organization,  and  cross-skill  work  permits  person¬ 
nel  to  be  shifted  to  areas  of  high  loadings  to  minimize  bottle¬ 
necks  and  equipment  downtimes.  Cross-skill  maintenance  also 
constitutes  a  form  of  OJT  to  enhance  maintenance  organizations' 
adaptabilities  in  satisfying  the  higher  (and  perhaps  more  vari¬ 
able)  workloads  that  would  be  experienced  under  combat  conditions. 
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Since  neither  team  nor  cross-skill  maintenance  provide 
valid  bases  for  assessing  training  effectiveness,  reported 
maintenance  data  would  have  to  identify  their  occurrences,  so 
that  they  could  be  eliminated  from  the  analyses. 

A  method  for  assessing  the  overall  performance  of  organiza¬ 
tional  maintenance  units  was  suggested  in  a  recent  study  per¬ 
formed  for  the  Naval  Equipment  Training  Center.*  The  concept 
behind  this  approach  is  that  some  maintenance  actions  are  not 
successful  in  correcting  equipment  malfunctions  or  result  in 
damage  to  equipment.  Such  cases  are  defined  as  errors;  three 
types  of  errors  are  defined,  as  follows: 

•  Equipments  that,  in  fact,  are  functioning  correctly  are 
replaced; 

•  Equipments  that,  in  fact,  are  malfunctioning  are  not 
replaced;  and 

•  Repairs  are  performed  that  do  not  correct  malfunctions 
or  result  in  other  malfunctions. 

If  the  occurrence  of  these  errors  could  be  unambiguously  asso¬ 
ciated  with  the  actions  of  maintenance  personnel  they  might 
provide  a  measure  of  performance  of  a  maintenance  unit  as  a 
whole.  The  AMES  model  is  limited  to  assessing  the  impact  of 
such  errors  on  corrective  maintenance  of  the  organizational 
echelon  of  Naval  aviation.  Its  applicability  to  assessing  the 
effectiveness  of  particular  training  programs  is  discussed  in 
the  section  below  that  addresses  the  Aviation  3-M  system. 


*Gold,  David,  Bruce  Kleine,  Frank  Fuchs,  Sal  Ravo,  and  Kay 
Inaba ,  Aircraft  Maintenance  Effectiveness  Simulation  (AMES) 
Model;  Final  Report.  Technical  Report  NAVTRAEQUIPCEN 
77-D-0028-1,  Naval  Training  Equipment  Center,  Orlando,  FL 
23813,  1980  (draft) . 


Individuals  performing  maintenance  are  identified  by  name  only  in  Initial  hardcopy  forms  that  are  retainer 
by  local  units  for  a  limited  time. 
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Each  of  the  four  reporting  systems  is  evaluated  below,  in 
summary  form.  Table  1  presents  an  overview  of  the  characteris¬ 
tics  of  each  system.  Displays  and  discussions  of  the  forms  on 
which  maintenance  data  are  recorded  in  the  four  systems  are 
presented  in  the  Appendix. 


IV.  NAVAL  AVIATION  3-M  AND  AIR  FORCE  66-1/66-5  SYSTEMS 

The  Naval  Aviation  3-M  and  Air  Force  66-1/66-5  systems  are 
discussed  together  because  they  provide  nearly  similar  data. 

In  fact,  it  appears  as  though  the  data  structures  derive  from  a 
common  base.  The  description  and  evaluation  found  below  applies 
to  both  systems,  except  where  specific  differences  are  noted. 

The  discussion  is  phrased  in  terms  of  aircraft  systems.  How¬ 
ever,  other  aeronautical  systems  (missiles,  aircraft  support 
equipment,  etc.)  use  the  same  data  descriptors;  together,  these 
systems  account  for  the  bulk  of  equipment  holdings  of  both  the 
Air  Force  and  the  Naval  air  arm. 

In  discussing  these  systems,  it  is  useful  to  distinguish 
among  three  basic  functions  provided  by  maintenance  organiza¬ 
tions: 

•  Aircraft  support  services  (including  flight-line  and 
flight  operations  services); 

•  Look-phase  of  equipment  inspections; 

•  Corrective  maintenance  (including  the  repair  of  deficien¬ 
cies  noted  during  the  look-phase  of  the  inspections,  the 
repair  of  malfunctions  noted  at  other  times,  modifications, 
and  non-maintenance  work  that  supports  corrective 
maintenance) . 


A .  DATA  REPORTED 


The  types  of  data  reported  by  these  systems  provide  at 
least  a  potential  for  assessing  maintenance  performance  in  a 
manner  that  reflects  on  training  effectiveness.  That  is,  the 
data  address  the  questions  of  what  equipment  was  maintained, 
what  was  wrong,  what  was  done,  and  by  whom. 

1.  What  Equipment 

The  equipment  on  which  maintenance  is  performed  is  identi¬ 
fied  through  the  combination  of  the  weapon  end-item  and  the 
Work  Unit  Code  (WUC)  structure  defined  for  that  end  item,  as 
required  by  MIL-STD  780E  (AS)  (Military  Standard  Work  Unit 
Codes  for  Aeronautical  Equipment) .  With  the  detail  embodied 
in  WUC  structures,  the  end-item/WUC  combination  typically 
identifies  equipment  that  can  be  associated  with  a  narrow  skill 
area  related  to  specific  training  programs. 

2.  What  Was  Wrong 

Over  200  explanations  (with  assigned  code  values)  are  pro¬ 
vided  to  describe  the  nature  of  equipment  malfunctions  or  other 
reasons  for  undertaking  maintenance.  These  codes  serve  as  an 
initial  diagnosis  to  aid  the  maintenance  process  and,  as  possi¬ 
bly  amended,  a  final  explanation  of  the  reasons  for  maintenance 
in  historical  records.  Most  of  the  reasons  for  maintenance  can 
be  loosely  grouped  into  six  categories,  as  follows: 

•  Loss,  failure,  deterioration,  etc.,  of  parts  and  compo¬ 
nents; 

•  Degraded  performance  or  out-of-range  tolerance  of  test 
readings  of  subsystems  and  assemblies; 

•  Accident  or  other  damage; 

•  Maintenance  and  operator  errors; 

•  No  malf unction--no  reason  provided; 


•  No  malf unction--assembli.es/components  processed  for  other 
reasons  (e.g.,  removed  for  scheduled  maintenance  or  to 
facilitate  other  maintenance) . 


3.  What  Was  Done 

Describing  what  was  done  in  the  maintenance  of  equipment 
involves  two  considerations:  the  extent  of  a  maintenance  job 
that  is  documented  separately  (maintenance  action  or  maintenance 
task,  as  discussed  above)  and  what  the  maintenance  entailed. 

Two  coded  descriptors.  Type  Maintenance  (TM)  and  Action  Taken 
(AT),  are  employed  to  describe  what,  in  fact,  was  done  in  the 
course  of  maintenance.  (Examples  of  TM  are  general  support, 
unscheduled  maintenance,  salvage,  and  various  types  of  inspec¬ 
tions;  examples  of  AT  are  remove,  repair,  calibrate,  and  corro¬ 
sion  treatment.)  Taken  together,  the  AT  and  TM  descriptors 
allow  an  unambiguous  classification  of  all  maintenance  performed 
into  the  three  major  functions:  support  services,  look-phase 
of  inspections,  and  corrective  maintenance.  However,  the 
amount  of  detail  provided  regarding  the  nature  of  maintenance 
performed  varies  greatly  among  these  functions. 

The  distinction  between  "maintenance  action"  and  "mainte¬ 
nance  task"  was  presented  above  (maintenance  action  consists 
of  a  number  of  maintenance  tasks) .  In  these  reporting  systems 
a  maintenance  action  is  identified  through  the  assignment  of  a 
Job  Control  Number  ( JCN) .  Each  combination  of  TM  and  AT  per¬ 
formed  by  each  work  center  involved  in  a  maintenance  action 
comprises  a  maintenance  task;  all  maintenance  tasks  are  sepa¬ 
rately  documented  in  historical  records,  and  each  task  is 
related  back  to  the  maintenance  action  through  the  JCN. 

4.  By  Whom 

Both  Air  Force  and  Navy  maintenance  performed  by  organi¬ 
zational  and  intermediate  echelons  and  the  elements  of  both 
echelons  are  formed  into  WCs.  In  both  reporting  systems 


documentation  of  each  maintenance  task  identifies  a  performing 
work  center;  however,  differences  between  organizational  and 
intermediate  echelons  in  the  way  work  centers  are  formed  has 
an  impact  on  how  closely  WC  identification  can  be  associated 
with  skill  area  and  training. 

For  both  services,  intermediate  maintenance  WCs  are  related 
to  type  of  equipment  maintained  in  a  way  that  relates  to  skill 
and  training.  (Examples  of  intermediate-level  WCs  are  airframe 
structure,  jet  engine,  navigation  system.)  Organizational  main¬ 
tenance  WCs  are  organized  around  both  type  of  equipment  (simi¬ 
lar  to  the  intermediate  organization)  and  functions  performed. 
The  functionally  organized  centers  (e.g.,  flight  line,  base 
flight,  inspection,  support  equipment)  are  associated  with  sup¬ 
port  services  and  the  look-phase  of  inspections,  and  these  WCs 
may  be  manned  by  personnel  with  a  variety  of  skills  and  training 
so  that  performance  would  be  difficult  to  relate  to  particular 
training  courses. 

In  the  Aviation  3-M  system,  initial  (hardcopy)  documenta¬ 
tion  of  corrective  maintenance  and  the  look-phase  of  major  in¬ 
spections  includes  names  (and  man-hours)  of  individuals  perform¬ 
ing  the  work.  While  this  might  provide  a  direct  relationship 
between  performance  and  training,  these  records  are  held  only 
by  the  originating  maintenance  units  and  are  retained  for  only 
6  months  so  that  their  availability  for  analyses  is  limited. 

B.  ASSESSMENT 

Even  though  the  reported  data  address  the  proper  questions 
for  assessing  training  effectiveness,  both  the  Aviation  3-M  and 
66-1/66-5  systems  have  limited  capabilities  in  this  regard.  The 
structures  of  the  five  data  codes  show  a  heavy  emphasis  on  cor¬ 
rective  maintenance.  (Three  of  the  five  categories  are  appli¬ 
cable  only  to  this  function,  as  shown  in  Table  2).  The  result 
is  that  the  usefulness  of  these  data  in  assessing  training 
differs  markedly  among  the  three  types  of  maintenance. 


TABLE  2.  EMPHASIS  OF  DATA  CODES 


;  Type  of  Maintenance  I 

Question 

Data  Code 

Support 

Services 

Look-Phase  of 
Inspections 

Corrective 

What 

equipment? 

Work  Unit  Code  (WUC) 

X 

What  was 
wrong? 

Malfunction  Code 

. 

X 

What 

Action  Taken  Code  (AT) 

X 

was 

done? 

Type  Maintenance  Code  (TM) 

X 

X 

x 

Who  did 
it? 

Work  Center  Code  (WC) 

X 

x 

1  * 

X 

1.  Support  Services  and  Look-Phase  of  Inspections 
By  definition,  support  services  and  the  look-phase  of  inspec¬ 
tions  do  not  involve  malfunctions.  Both  functions  are  typically 
concerned  with  complete  end-items,  so  that  the  WUC  has  little 
meaning.  The  AT  code  is  used  only  with  corrective  maintenance. 
Nominally,  this  leaves  the  combination  of  TM  and  the  performing 
WC  as  the  only  way  to  characterize  both  functions,  and  few  TM 
code  values  are  defined.  (Only  one  value  of  the  TM  code  is 
provided  to  cover  all  support  services;  TM  values  describing 
inspections  are  limited  to  the  type  of  inspection  performed.) 

This  sketchy  specification  is  supplemented  by  the  use  of 
special  codes  that  are,  in  effect,  extensions  of  and  partially 
redundant  with,  the  TM  code.  In  the  Aviation  3-M  system,  Sup¬ 
port  Action  Codes  (SAC)  are  defined  which  identify  (1)  several 
types  of  support  services  (e.g.,  operational  support,  corrosion 
control),  (2)  different  types  of  maintenance  support  operations 
(e.g.,  shop  support,  engine  build-up  and  tear-down);  and  (3) 
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types  of  minor  inspections.  For  major  inspections,  the  WUC 
block  of  the  maintenance  report  form  is  used  with  a  special 
code  to  identify  the  type  of  inspection  performed.  In  the 
66-1/66-5  system  a  Support  General  code  structure  is  defined, 
and  its  values  are  inserted  in  the  WUC  block  of  the  reporting 
form  to  identify  the  type  of  support  service  provided  or  inspec¬ 
tion  performed  (similar  to  the  SAC  and  special  use  of  the  WUC 
block  in  the  3-M  system) . 

In  fact,  a  number  of  different  tasks  are  involved  in  sup¬ 
port  services  while  the  look-phase  of  some  types  of  inspections 
are  extensive  undertakings.  In  both  cases,  a  range  of  skills 
(and  training)  may  be  involved.  While  the  data  code  values 
available  may  be  adequate  for  managing  these  functions,  they 
seem  wholly  insufficient  to  characterize  the  different  tasks 
in  a  way  that  identifies  the  skills  employed,  a  necessary  con¬ 
dition  for  assessing  training  effectiveness. 

It  might  be  argued  that  some  measure  of  maintenance  per¬ 
formance  could  be  obtained  from  the  combination  of  TM  (as  sup¬ 
plemented  by  the  SAC  and  special  use  of  the  WUC  block)  and  iden¬ 
tification  of  work  centers  or  individuals.  Since  both  appear 
to  involve  standardized  tasks  that  are  essentially  unchanged 
between  different  organizations  maintaining  the  same  equipment 
end-items,  an  essential  ingredient  for  comparability  is  present. 

However,  it  does  not  appear  valid  to  relate  performance 
measured  in  this  way  to  training  effectiveness.  Support  serv¬ 
ices  and  minor  inspections  are  provided  by  organizational  eche¬ 
lon  WCs  that  specialize  in  these  functions.  For  maintenance 
managed  within  the  Aviation  3-M  and  66-1  systems  neither  func¬ 
tion  represents  a  skill  area  associated  with  a  particular  type 
of  equipment  (such  as  avionics  or  engines)  that  can  be  identi¬ 
fied  with  particular  training  programs,  and  one  might  expect 
to  find  such  WCs  populated  with  individuals  from  a  variety  of 
training  backgrounds.  Within  the  66-5  management  system  these 
functions  are  performed  by  personnel  specifically  trained  in 
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support  services  as  well  as  personnel  trained  in  specific  tech¬ 
nical  skill  areas.  With  respect  to  major  inspections,  the 
inspection  WCs  may  be  assisted  by  personnel  from  repair  WCs 
(either  organization  or  intermediate  echelon)  that  can  be  asso¬ 
ciated  with  specific  skill  areas  and  training.  This  is  neces¬ 
sarily  team  maintenance,  and  there  appears  to  be  no  way  to 
separate  the  contributions  of  different  individuals  or  work 
centers  to  the  overall  effort. 

Assessing  training  effectiveness  in  this  manner  would  re¬ 
quire  a  large  sample  of  similarly  described  maintenance  tasks 
across  organizations  (maintaining  the  same  type  of  equipment) 
that  could  be  differentiated  solely  on  the  basis  of  training 
received.  Even  with  large  samples,  inferences  relating  per¬ 
formance  to  training  effectiveness  would  be  weak.  Both  service 
and  inspection  look-phases  are  predominantly  team  efforts  where 
the  impact  of  newly  trained  individuals  on  overall  performance 
would  be  diluted  by  the  supervision  and  abilities  of  experienced 
personnel.  A  further  consideration  in  drawing  conclusions  is 
that  the  skills  involved  in  support  service  and  inspections 
may  be  relatively  small  and  unrepresentative  of  the  range  of 
skills  for  which  individuals  are  trained. 

2.  Corrective  Maintenance 

At  first  appearance  this  is  the  more  promising  area  for 
determining  maintenance  performance  in  a  way  that  reflects  on 
training  effectiveness.  The  types  of  maintenance  reported 
include,  essentially,  the  full  range  of  skills  for  which  people 
are  trained;  the  full  range  of  data  codes  are  available  for 
describing  both  the  nature  of  maintenance  and  who  performed  it. 
If  it  is  accepted  that  the  combination  of  repair  WC  and  work 
unit  code  adequately  describes  skill  areas  that  can  be  related 
to  training  programs,  then  the  TM,  AT,  and  malfunction  codes 
are  available  to  identify  what  a  maintenance  task  involves  in 
terms  of  skill  and  effort. 
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In  the  Aviation  3-M  system,  approximately  200  malfunction 
codes  are  defined;  twelve  TM  codes  and  22  AT  codes  (including 
10  that  provide  reasons  why  no  action  was  taken)  are  applicable. 
These  values  allow  a  set  of  close  to  30,000*  unique  combinations 
of  the  three  descriptors  for  a  given  WUC  and  the  set  will  vary 
among  major  systems  of  a  weapon  end-item.  In  the  66-1/66-5  sys¬ 
tems  something  on  the  order  of  50,000  combinations  appear  pos¬ 
sible;  they  define  roughly  20  percent  more  malfunction,  10  per¬ 
cent  more  action  taken,  and  30  percent  more-type  maintenance 
descriptors  than  Aviation  3-M. 

This  large  number  of  combinations  presents  its  own  problem. 
One  must  consider  whether  the  systems  are  too  complex  to  allow 
reliable  groupings  and  separations  of  similar  and  different 
maintenance  tasks.  There  is  no  doubt  that  the  number  of  poten¬ 
tial  combinations  can  be  significantly  reduced  and  still  permit 
one  to  identify  similar  maintenance  tasks.  Roughly  one-half  of 
the  malfunction  codes  are  applicable  to  each  major  system  (de¬ 
noted  by  the  first  two  digits  of  the  WUC),  e.g.,  less  than  90 
values  are  applicable  to  landing  gear  systems.  l'ome  combina¬ 
tions  of  malfunction,  TM,  and  AT  codes  are  indicative  of  the 
same  maintenance  task,  e.g.,  it  is  irrelevant  whether  a  repair 
is  made  as  unscheduled  maintenance  or  as  the  fix  phase  of  an 
inspection.  Even  if  90  percent  could  be  eliminated  on  such 
grounds,  there  remain  a  large  number  of  combinations  to  be  in¬ 
terpreted,  especially  when  their  full  meanings  might  differ  as 
a  function  of  other  data  contained  in  the  maintenance  records. 
For  example,  the  skills  required  to  troubleshoot  a  particular 
model  of  electronic  equipment  with  the  same  apparent  (or  stated) 
malfunction  may  be  quite  different  between  organizational  and 
intermediate  echelon  maintenance.  At  the  same  time,  one  must 
consider  whether  the  number  of  logical  combinations  of  codes 
that  are  available  within  the  systems  is  sufficient  to  distin¬ 
guish  among  all  the  different  maintenance  requirements  on  such 


*200  x  12  x  (  22  -  9)  . 
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complex  equipments  as  military  aircraft.  Both  questions  are 
crucial  in  assessing  the  reporting  systems,  but  we  can  offer 
no  definitive  answers. 

There  are,  no  doubt,  a  number  of  standardized  and  repeti¬ 
tively  performed  maintenance  tasks  that  can  be  unambiguously 
identified  from  the  values  given  to  one  (or  more)  of  the  three 
data  codes  and  thereby  serve  as  indicators  of  maintenance  per¬ 
formance  in  studies  of  training  effectiveness.  For  example, 
removing  or  installing  a  particular  piece  of  equipment  on  a 
given  end-item  (an  action  taken)  requires  the  same  skills,  re¬ 
gardless  of  why  the  removal  or  installation  was  required,  i.e., 
regardless  of  malfunction  or  TM  involved.  Limiting  assessment 
to  such  cases,  however,  amounts  to  evaluation  by  exception  and 
raises  again  the  question  of  whether  the  set  of  tasks  that  can 
be  identified  in  this  manner  is  representative  of  the  range  of 
skills  addressed  by  training  programs. 

In  fact,  the  assumption  that  the  combination  of  WUC  and  WC 
adequately  define  skill  areas  and  training  may  not  be  valid  for 
reasons  that  have  to  do  with  unit  operational  requirements 
( rather  than  the  nature  of  reporting  systems' .  Maintenance  in 
the  Tactical  Air  Forces  (TAC,  PACAF,  AAC,  USAFE)  is  managed 
through  the  66-5  system  (referred  to  as  Production-Oriented 
Maintenance  Organization,  or  POMO) .  Maintenance  squadrons  are 
planned  to  be  deployed  with  flight  squadrons  ( in  the  event  of 
mobilization)  and  are  structured  with  the  objective  of  sustain¬ 
ing  high  levels  of  operations  (sortie  generation)  in  the  deployed 
condition.  Such  operating  levels  should  require  a  significant 
amount  of  cross-skill  maintenance;  combat  readiness  in  these 
organizations  would  require  cross-skill  maintenance  to  be  em¬ 
ployed  during  non-deployed  periods.  This  appears  to  be  accom¬ 
plished  in  two  ways,  (1)  by  forming  permanent  repair  teams  com¬ 
posed  of  individuals  with  diverse  skill  and  training  backgrounds 
and  attaching  these  teams  to  a  single  WC  and  ( 2)  by  assigning 
repair  teams  or  individuals  from  one  skill  area  to  maintenance 
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tasks  controlled  by  WCs  associated  with  other  skills.  For  SAC 
and  MAC  units,  whose  maintenance  is  managed  through  the  66-1 
system  and  with  the  Navy  maintenance  squadron  organization, 
cross-skill  maintenance  may  occur  to  varying  extents  as  the 
result  of  differences  in  local  command  policies  and  variations 
in  workloads. 

In  any  case,  a  consistent  relationship  between  WC,  WUC,  and 
skill  area  in  reported  maintenance  data  cannot  be  established, 
and  this  weakens  the  argument  for  using  maintenance  data  to  de¬ 
termine  training  effectiveness.  To  establish  such  a  relation¬ 
ship,  data  would  have  to  identify  where  (on  what  individual 
maintenance  tasks)  cross-skill  work  occurs,  so  that  these  cases 
could  be  eliminated  from  the  sampling  process;  it  is  uncertain 
that  this  is  possible. 

Team  maintenance  introduces  a  further  complication  in  in¬ 
ferring  training  effectiveness.  It  appears  to  be  a  common,  if 
not  prevalent,  practice,  especially  in  Air  Force  units  where 
recent  trainees  must  undergo  some  period  of  OJT  and  be  expressly 
qualified  before  they  can  perform  independent  work.  Air  Force 
reporting  notes  the  occurrence  of  team  maintenance  so  that,  at 
the  least,  its  extent  can  be  known,  and  it  may  be  possible  to 
characterize  the  types  of  maintenance  tasks  that  are  typically 
performed  by  a  single  individual.  In  the  Aviation  3-M  system, 
only  the  initial  hardcopy  records  make  any  note  of  the  numbers 
of  persons  responsible  for  a  maintenance  task  so  it  would  be  im¬ 
possible  to  determine  either  the  extent  or  occurrences  of  team 
maintenance  from  the  ADP  files  that  contain  the  bulk  of  histori¬ 
cal  data. 
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3.  Maintenance  Errors 


Gold,  et  al.*  define  three  types  of  maintenance  errors  and 
cite  the  following  Aviation  3-M  data  as  evidence  of  their  occur¬ 
rence  . 


Error 


Evidence 


1.  Replacing  equipments  that 
are  functioning  correctly. 


2.  Not  replacing  equipments 
that  are  malfunctioning. 


Equipment  removed  at  organiza¬ 
tional  maintenance  shows  no 
defects  when  it  is  bench-checked 
at  intermediate  maintenance. 

(One  malfunction  code  identi¬ 
fies  no-defect  items.) 

Successive  flights  of  one  air¬ 
craft  are  accompanied  by  correc¬ 
tive  maintenance  actions  on  the 
same  aircraft  system,  and  these 
actions  suddenly  stop.  That  is, 
the  final  corrective  action  was 
successful.  Analysis  of  the 
documentation  of  the  series  of 
corrective  actions  will  deter¬ 
mine  whether  the  repetitive 
failures  resulted  from  error  or 
other  causes. 


♦Gold  et  al..  Aircraft  Maintenance  Effectiveness  Simulation 
(AMES)  Model:  Final  Report,  Technical  Report  NAVTRAEQUIPCEN 
77-D-0028-1,  Naval  Training  Equipment  Center,  Orlando,  FL 
32813  (1980) . 
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3.  Repairs  do  not  correct  The  occurrence  of  15  malfunction 

malfunctions  or  result  codes  in  3-M  records  is  evidence 

in  other  malfunctions.  of  either  definite  errors  or 

probable  errors.  For  those  codes 
that  are  classed  as  probable 
errors,  analysis  of  prior  main¬ 
tenance  actions  will  determine 
whether  the  malfunction  resulted 
from  error  or  other  causes. 

The  impact  of  errors  at  the  organizational  level  was  esti¬ 
mated  by  applying  one  year*  s  worth  of  3-M  records  for  six  Navy 
aircraft  squadrons  to  the  AMES  model.  The  simulation  concluded 
that,  with  a  90  percent  reduction  in  error  rates,  the  same  number 
of  missions  could  b.-  flown  with  one  fewer  aircraft  per  squadron 
of  12  aircraft.  Numbers  of  missions  flown  is  only  one  of  sev¬ 
eral  simultaneous  measures  of  unit  performance  among  which 
trade-offs  can  be  made  (e.g.,  maintenance  man-hours  per  flying 
hour,  operational  readiness,  cannibalization  rates),  and  no 
other  measures  were  addressed  in  the  simulation. 

Since  evidence  of  maintenance  errors  is  found  in  the  Avia¬ 
tion  3-M  records,  the  data  are  capable  of  characterizing  the 
tasks  on  which  they  occur  (i.e.,  the  equipment  involved,  what 
was  wrong  with  it,  what  was  done  to  it,  and  by  whom).  Further, 
with  the  similarities  in  data  developed  in  the  Aviation  3-M  and 
66-1/66-5  systems,  each  should  be  equally  capable  of  relating 
errors  to  training  effectiveness.  However,  there  are  several 
considerations  beyond  the  data  reported  that  must  be  addressed 
before  a  relationship  between  these  errors  and  training  effec¬ 
tiveness  can  be  established,  as  follows: 

•  Whether  the  three  events  that  are  classified  as 
errors  do,  in  fact,  constitute  errors,  and,  if 
so,  whether  they  can  unambiguously  be  associated 
with  the  maintenance  function; 
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•  Whether  these  events  occur  with  sufficient  fre¬ 
quency  across  the  range  of  aviation  equipments 
and  skill  areas  for  whicn  personnel  are  trained; 
and 

•  Whether  the  tasks  on  which  they  occur  (in  an 
individual  skill  area)  are  representative  of 
the  range  of  tasks  for  which  personnel  are 
trained  in  that  skill. 

The  concept  of  error  and  the  3-M  data  cited  as  evidence 
have  been  reviewed  with  personnel  from  the  Naval  Air  Systems 
Command  (NAVAIR)  and  the  Deputy  Chief  of  Naval  Operations 
(DCNO)  for  Air  Warfare  that  are  experienced  in  both  aircraft 
maintenance  and  the  Aviation  3-M  system.  In  all  cases,  these 
individuals  stated  that  replacements  of  operating  equipment, 
failure  to  replace  malfunctioning  equipment,  and  most  of  the 
15  particular  malfunction  codes  can  be  consistent  with  efficient 
unit  operations  (i.e.,  they  are  not  necessarily  errors).  They 
further  stated  that  whether  or  not  these  events  are  considered 
as  errors,  other  aspects  of  military  operations  may  be  as  or 
more  important  than  maintenance  as  reasons  for  their  occurrence. 
The  other  aspects  that  were  cited  include  the  complexity  and 
reliability  of  equipment,  the  adequacy  of  the  support  equipment 
and  documentation  provided  organisational  maintenance  personnel, 
the  competance  of  equipment  operating  personnel,  unit  operating 
environments,  and  command  decisions. 

A  frequently  cited  example  was  that  the  support  equipment 
and  documentation  for  complex  avionics  systems  provided  to  organ¬ 
izational  maintenance  are  frequently  incapable  of  isolating 
malfunctions  to  a  single  black  box  within  the  system.  If  squad¬ 
ron  policy  is  to  replace  and  bench-check  one  box  at  a  time,  an 
incorrect  first  guess  as  to  which  box  to  replace  will  result  in 
two  "errors"  --  replacing  functioning  equipment  and  not  replac¬ 
ing  malfunctioning  equipment;  if  squadron  policy  is  to  replace 
the  total  system,  the  result  will  be  the  replacement  of  a  series 


of  functioning  equipments.  Deficiencies  of  support  equipment 
and  documentation  might  be  considered  as  a  maintenance  system 
inefficiency,  but  it  is  not  a  valid  reflection  of  the  capabili¬ 
ties  of  maintenance  personnel.  In  addition,  it  may  be  that  the 
cost  ct  providing  organizational  maintenance  units  with  test 
equipment  that  would  resolve  all  such  ambiguities  is  higher  than 
the  cost  of  the  erroneous  equipment  replacements. 

The  impacts  of  these  other  facets  of  unit  operations  may 
be  systematic  between  different  squadrons  operating  the  same 
equipments  so  that  the  frequency  of  occurrence  of  errors  will 
not  provide  a  valid  basis  for  comparison,  e.g.,  carrier-  versus 
land-basing,  carrier-based  and  operating  in  different  theatres, 
and  different  squadron  command  practices.  Verifying  that  such 
purported  errors  are,  in  fact,  errors,  isolating  their  causes 
(e.g.,  maintenance,  test  procedures,  operating  environment), 
and  establishing  comparability  among  different  units  appears  to 
involve  extensive  data  and  analysis,  possibly  including  material 
not  currently  reported  through  Aviation  3-M.  Considering  the 
size  of  the  Naval  air  arm,  it  might  prove  difficult  or  impossible 
to  gather  a  sufficiently  large  sample  to  identify,  with  confi¬ 
dence,  true  errors  that  can  be  attributed  to  maintenance  person¬ 
nel,  even  if  the  impacts  of  other  asoects  of  unit  operations 
were  not  systematic. 

Further  complications  in  assessing  training  effectiveness 
through  analysis  of  errors  lie  in  whether  the  events  that  can 
be  verified  as  errors  occur  across  the  range  of  different  air¬ 
craft  equipments  (and  associated  skill-areas)  and,  for  a  given 
type  of  equipment,  whether  the  types  of  tasks  on  which  they 
occur  are  representative  of  the  job  requirements  of  the  associ¬ 
ated  skill-areas.  Some  types  of  equipments  are  difficult  to 
replace  because  of  size  or  structural  integration  (e.g.,  en¬ 
gines  and  flight  control  systems)  while  other  types  are  compat¬ 
ible  with  and  designed  for  easy  replacement  (e.g.,  avionics). 


Efficient  maintenance  practice  would  dictate  that  a  dispropor¬ 
tionate  share  of  off-equipment  maintenance  would  occur  in  areas 
such  as  avionics  so  that  these  would  be  the  skill-areas  where 
errors  involving  removals  would  be  concentrated.  NAVAIR  has 
previously  studied  faulty  removals  of  equipment  and  found  that 
90  percent  occur  in  avionic  systems.  This  leaves  little  ground 
for  commission  of  this  type  of  error  in  other  systems  and  casts 
doubt  upon  the  capability  of  error  analysis  for  assessing  train¬ 
ing  effectiveness  across  a  range  of  skill  areas.  On-equipment 
maintenance  and  equipment  replacement  is  concentrated  at  the  or¬ 
ganizational  echelon  so  that  all  errors  involving  removals  of 
equipment  would  occur  at  this  level,  but  training  programs  are 
not  necessarily  limited  to  organizational  maintenance  tasks. 
Further,  it  cannot  be  assumed  that  the  malfunctions  that  result 
in  equipment  replacement  (as  distinct  from  those  that  result  in 
on-equipment  corrections)  utilize  the  range  of  skills  in  which 
organizational  maintenance  personnel  are  trained. 

The  arguments  and  examples  above  address  only  the  two 
classes  of  "errors"  that  involve  replacements  of  equipment. 
However,  the  discussions  suggested  that  similar  arguments  would 
apply  to  the  15  specified  malfunction  codes.  Each  of  the  dis¬ 
cussants  singled  out  a  few  of  the  15  codes  as  being  "primarily" 
maintenance  induced,  but  there  was  little  agreement  concerning 
which  of  the  15  fell  into  this  category.  in  addition,  they 
typically  offered  other  primary  causes  for  a  number  of  remaining 
codes,  implied  that  individual  codes  would  be  primarily  associ¬ 
ated  with  a  limited  number  of  aircraft  system  types,  and  that 
the  main  underlying  causes  (support  equipment,  operational 
environment,  maintenance)  lying  behind  particular  codes  may 
differ  according  to  the  type  of  aircraft  system.  When  all  the 
arguments  are  considered,  it  appears  highly  unlikely  that  the 
type  of  analysis  proposed  by  AMES  would  shed  light  on  the  rela¬ 
tive  effectiveness  of  training  programs. 
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4 .  Conclusion 

The  basic  problem  in  assessinj  training  effectiveness  in 
the  field  may  lie  in  the  complexity  of  military  equipment  and 
organization.  Aviation  3-M  and  66-1/66-5  are  large  and  complex 
systems  that  are  concerned  with  management  and  control  of  main¬ 
tenance  at  the  unit  level.  Design  and  implementation  of  a  data 
system  that  would  satisfy  the  management  function  and  also  pro¬ 
vide  assessments  of  the  performance  and  training  of  individuals 
does  not  seem  feasible.  Assessing  the  effectiveness  of  mainte¬ 
nance  training  would  appear  to  require  a  separate  data  collec¬ 
tion  and  processing  system.  However,  with  current  organiza¬ 
tional  practices  (e.g.,  cross-skill  and  team  maintenance)  it 
is  hard  to  see  how  a  data  system  specifically  addressed  to  the 
performance  of  maintenance  personnel  could  generate  adequate 
information  regarding  training  effectiveness  from  day-to-day 
operations  of  maintenance  units. 

This  rather  discouraging  picture  has  been  confirmed  infor¬ 
mally  by  discussions  with  personnel  in  NAVAIR  and  the  Air  Force 
Deputy  Chief  of  Staff  for  Logistics.  Their  view  is  that  these 
systems  cannot  answer  questions  regarding  training  effectiveness. 
In  fact,  NAVAIR  currently  commits  resources  tc  determining  unit 
effectiveness  by  further  analysis  of  3-M-gener..ted  records--the 
NAMP  Improvement  Program  (NIP).  In  this  case,  unit  performance 
is  evaluated  through  ADP  in  terms  of  gross  criteria,  such  as 
operational  readiness  and  cannibalization  rate^,  assessing 
reasons  for  the  observed  performance  and  prescriptions  for 
improvement  require  direct  contact  (such  as  field  visits)  with 
the  units  evaluated.  At  that,  the  approach  is  more  straight¬ 
forward  and  assessment  criteria  are  certainly  more  visible  than 
would  be  the  case  in  evaluating  the  performance  of  individuals 
trained  in  one  of  the  numerous  skills  required  in  a  maintenance 
organization. 
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V.  SHIPS'  3-M 


Naval  ship  maintenance  is  managed  through  the  Ships'  Main¬ 
tenance  and  Material  Management  (Ships'  3-M)  System.  Ships, 
their  outfitting,  and  their  operation  display  features  not  found 
in  other  weapon  systems  that  affect  the  way  maintenance  is  per¬ 
formed  and  both  the  extent  and  format  of  reported  maintenance 
data. 

Ships  of  the  same  class  ( including  those  launched  within 
the  same  time  period)  generally  display  configuration  differ¬ 
ences,  especially  with  regard  to  the  smaller  assemblies  and  com¬ 
ponents  comprising  major  functional  systems.  For  example,  while 
the  use  of  particular  models  of  the  major  elements  of  a  ship's 
main  propulsion  system  (boiler,  turbine,  etc.)  may  be  a  design 
specification,  ship  builders  are  free  to  use  alternative  models 
of  valves  and  pumps  so  long  as  they  are  compatible  with  the 
major  elements  of  the  system.  The  models  installed  initially 
may  vary  between  ships  produced  under  the  same  contract,  and 
the  initial  outfitting  may  be  replaced  by  different  models  dur¬ 
ing  ships'  lifespans.  Such  variations  complicate  problems  in 
configuration  control  and  stockage  of  repair  parts. 

Ships  are  capable  of  long  periods  of  independent  operations 
where  a  ship's  force  (crew)  provides  the  only  source  of  mainte¬ 
nance  of  the  ship's  material  condition.  However,  ships  are  not 
outfitted  or  manned  to  be  capable  of  maintaining  full  operational 
capabilities,  in  all  assigned  missions,  at  all  times.  Ship  oper¬ 
ations,  lack  of  repair  part  stocks,  and  the  unpredictable  occur¬ 
rence  of  equipment  breakdowns  may  require  that  some  maintenance 
be  deferred  for  later  correction  by  ship's  force;  the  ship  incurs 
degraded  capabilities  during  these  time  periods.  In  addition, 
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ships  are  not  outfitted  or  manned  to  be  wholly  self-sufficient 
in  terms  of  maintenance.  Some  equipment  repairs  may  require 
facilities,  tools,  and  documentation  not  available  to  the  ship's 
force  or  be  beyond  their  skills.  In  these  cases,  maintenance 
must  be  deferred  until  the  ship  is  provided  an  "availability" 
at  an  Intermediate  Maintenance  Activity  (IMA)--a  tender  or  ship¬ 
yard. 

Maintenance  documentation  under  Ships'  3-M  reflects  the 
shipboard  environment  and  higher-command  responsibilities  for 
tracking  and  control  of  fleet  readiness  and  ship  configurations 
and  the  management  of  IMA  workloads.  As  a  general  rule  (subject 
to  two  exceptions  noted  below) ,  reporting  of  maintenance  is  re¬ 
quired  only  where  changes  in  ship  configurations  result  and 
where  deferral  periods  exceed  30  days.  (The  30-day  period 
applies  to  both  preventive  and  corrective  maintenance.)  The 
two  exceptions  are  that  all  corrective  maintenance  on  submarines 
is  reported  and  all  maintenance  on  equipments  placed  on  the 
Selected  Equipment  List  ( SEL)  is  reported.  (The  SEL  consists 
of  roughly  500  equipments  with  histories  of  maintenance  problems 
or  equipments  that  are  new  or  modified  and  for  which  maintenance 
profiles  and  parts  stockage  requirements  have  not  been  developed.) 

A.  DATA  REPORTED 

1.  What  Equipment 

The  form  in  which  data  are  reported  in  ships'  3-M  does  not 
answer  the  basic  questions  of  What  equipment?  What  was  wrong 
with  it?  What  was  done  to  it?  and  By  whom?  in  a  manner  that 
sheds  light  on  maintenance  performance  or  training  effectiveness. 
Equipment  undergoing  maintenance  is  identified  through  two  sys¬ 
tems:  the  Equipment  Identification  Code  (EIC)  and  the  Allowance 

Parts  List  ( APL) .  The  EIC  is  structured  to  identify  functions 
performed  by  equipment.  In  a  four-character  code,  the  first 
character  denotes  major  functional  areas  (e.g.,  electric  power 


generation,  main  propulsion),  and  the  other  three  characters 
identify  functions  performed  with  the  major  areas  (e.g.,  power 
generation  control  group  or  system,  main  propulsion  gas  turbine.) 
A  single  El C  value  typically  includes  a  conglomerate  of  smaller 
equipments  (such  as  all  the  meters  and  controls  associated  with 
a  control  panel)  which  may  malfunction  and  require  maintenance. 
Except  in  the  areas  of  electronic  and  ordnance  equipment,  a 
single  EIC  code  value  refers  strictly  to  the  function  performed 
and  may  encompass  a  large  number  of  equipment  configurations 
(equipment  models  and  manufacturers).  For  electronic  and  ord¬ 
nance  equipment,  a  single  EIC  value  refers  to  both  the  function 
performed  and  the  specific  configuration  of  an  equipment  end- 
item  ( in  terms  of  standard  AN  and  Mk/Mod  number  designations) . 

In  neither  case,  though,  is  the  EIC  structured  to  identify  com¬ 
ponents  of  the  end-item  (such  as  power  supplies,  displays). 

That  is,  ohe  EIC  value  references  the  complete  system  and  not 
the  elements  of  the  system  that,  in  fact,  fail  and  undergo 
maintenance. 

The  APL  is  a  technical  document  used  in  equipment  mainte¬ 
nance  and  ship  configuration  control.  One  APL  refers  to  a  par¬ 
ticular  model  of  an  end-item  or  major  assembly  produced  by  a 
single  manufacturer  and  associated  with  a  unique  parts  list.  A 
number  of  APLs  may  be  associated  with  a  single  EIC  value  since 
a  number  of  end-item  models  may  provide  a  particular  function 
and  because  a  number  of  manufacturers  may  produce  a  single  end- 
item  model.  One  section  of  an  APL  consists  of  a  parts  list  (to 
the  bit  and  piece  level)  in  terms  of  national  stock  number  and 
manufacturer's  part  number.  Like  the  EIC  system,  APLs  typically 
provide  identification  only  to  the  system  or  major  assembly 
level. 
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without  any  defined  set  of  keywords.  While  this  allows  great 
flexibility  of  description,  it  leaves  no  way  by  which  similarity 
or  comparability  among  maintenance  actions  can  be  established. 
Comparability  is  further  complicated  by  the  formation  of  only 
one  maintenance  record  for  reporting  a  complete  maintenance 
action,  rather  than  well-defined  individual  (and  repetitive) 
tasks  that  are  a  part  of  numerous  maintenance  actions.  Given 
the  extent  and  complexity  of  maintenance  actions,  it  could  well 
be  that  no  two  are,  in  fact,  sufficiently  similar  to  provide  a 
basis  for  comparison. 

3 .  By  Whom 

The  identification  of  who  performs  maintenance  provides 
little  in  the  way  of  useful  information  for  assessing  training 
effectiveness,  although  report  forms  may  identify  both  individ¬ 
uals  and  WCs.  Individuals  are  named  as  contact  points  rather 
than  as  those,  in  fact,  performing  the  maintenance.  More  than 
one  WC  may  be  involved  in  the  completion  of  maintenance,  and 
there  is  no  way  to  separate  the  contributions  of  (or  tasks  per¬ 
formed  by)  the  different  centers.  Further,  when  maintenance  is 
deferred,  it  is  frequently  because  the  facilities,  skills,  and/ 
or  materials  required  are  available  only  at  IMAs.  In  these 
cases,  the  completion  of  maintenance  will  generally  involve 
personnel  from  both  the  ship  undergoing  repair  and  the  IMA. 

B.  Assessment 

The  Ships'  3-M  system  appears  to  offer  no  potential  for 
assessing  maintenance  performance  of  individuals.  Reasons  for 
this  negative  conclusion  lie  in  considerations  as  basic  as  the 
nature  of  shipboard  equipment  and  ship  operations  rather  than 
simply  features  of  the  reporting  system  itself. 
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Regardless  of  the  form  or  extent  of  maintenance  reporting, 
the  lack  of  standardization  of  ship  outfitting  is  a  sufficient 
reason  to  conclude  that  relationships  cannot  easily  be  estab¬ 
lished  between  the  reported  data  and  the  capabilities  of  main¬ 
tenance  personnel  (and,  hence,  training  effectiveness).  While 
ships  of  the  same  class  may  be  outfitted  to  the  same  mission 
capabilities,  the  systems  by  which  those  capabilities  are  at¬ 
tained,  their  assemblies  and  components,  and  their  on-board 
locations  may  differ.  These  differences  exist  at  the  time  of 
initial  ship  acceptance  and  increase  with  age  as  ships  undergo 
overhaul,  modernization,  and  other  configuration  changes.  Seem¬ 
ingly  minor  differences  in  equipment  (e.g.,  substitute  models 
of  valves  used  in  high-pressure  steam  lines)  and  location  of 
the  same  equipment  (e.g.,  engine  room  or  upper  deck  location 
of  a  particular  configuration  of  air-compressing  and  drying 
equipment)  may  have  significant  impacts  on  equipment  perfor¬ 
mance  and  maintainability.  While  these  impacts  will  be  re¬ 
flected  in  the  reported  maintenance  data,  they  cannot  be  taken 
as  indicators  of  the  capabilities  of  maintenance  personnel  or 
of  training  effectiveness.  Even  though  ships'  equipment  con¬ 
figurations  might  be  exactly  known,  they  may  be  sufficiently 
different  that  adequate  samples  of  comparable  maintenance 
actions  could  not  be  gathered. 

A  further  complication  involves  ship  overhaul  periods. 
Frequency  of  equipment  breakdown  appears  to  increase  as  a  func¬ 
tion  of  the  time  since  last  overhaul,  and  there  are  reasons  to 
assume  that  the  amount  of  effort  to  effect  repair  will  vary  in 
a  similar  fashion.  At  any  given  time,  there  will  be  a  distribu¬ 
tion  of  times  since  overhaul  among  ships  of  the  fleet,  and  this 
should  be  reflected  in  reported  maintenance  data.  To  the  extent 
that  time  since  overhaul  does  have  an  impact  on  maintenance  per¬ 
formance  criteria  (e.g.,  breakdown  rates,  man-hours  to  repair) 
it  will  mask  the  effect  of  training  effectiveness  on  performance 
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A  most  telling  shortcoming  of  the  management  system  is  its 
incomplete  documentation.  Both  preventive  and  corrective  main¬ 
tenance  should  be  completed  without  deferral  whenever  possible. 
Even  though  it  is  not  possible  to  quantify  the  fraction  of  main¬ 
tenance  effort  reported,  it  seems  probable  that  the  major  por¬ 
tion  of  surface  ship  maintenance  will  not  be  deferred  and,  hence, 
not  reported,  and  that  a  significant  portion  of  submarine  main¬ 
tenance  will  consist  of  non-deferred  preventive  maintenance  that 
also  will  not  be  reported.  In  addition,  it  can  be  assumed  that 
there  are  innate  differences  between  tasks  that  are  deferred 
and  those  that  are  not.  As  a  result,  it  cannot  be  assumed  that 
the  skills  required  for  that  maintenance  which  is  reported  will 
comprise  a  repre  sentative  sample  of  the  range  of  skills  for 
which  personnel  are  trained. 

A  second  major  shortcoming  of  the  system  is  that  the  indi¬ 
vidual  historical  records  that  are  developed  encompass  all  ef¬ 
forts  associated  with  a  complete  maintenance  action.  Mainte¬ 
nance  actions  are  large-scale  efforts  compared  with  individual 
maintenance  tasks  (that  form  the  individual  records  by  which 
aircraft  maintenance  is  reported) .  The  larger  the  scope  of 
effort  the  smaller  the  chance  that  maintenance  undertakings 
will,  in  fact,  be  comparable.  Indeed,  if  each  maintenance 
action  is  unique  in  some  way  there  will  be  little  or  no  grounds 
for  comparative  analysis.  (This  statement  might  hold  true  even 
if  all  maintenance  were  reported  and  if  ships  and  their  outfit¬ 
ting  were  wholly  standardized.)  In  addition,  when  one  considers 
that  descriptions  of  what  is  wrong  with  equipment  and  what  is 
done  to  repair  it  can  be  provided  only  in  English  phrases,  the 
probability  of  correctly  sorting  reported  maintenance  into  sets 
of  comparable  undertakings  appears  remote. 

Further,  such  large-scale  undertakings  would  likely  involve 
personnel  with  a  variety  of  skills  (and  training  backgrounds) 
from  several  WCs.  Where  maintenance  is  completed  with  IMA 
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assistance  personnel  from  at  least  two  WCs  would  be  involved-- 
one  from  the  tended  ship  and  one  from  the  IMA.  At  completion, 
man-hours  will  necessarily  have  been  expended  by  both  ship's 
force  and  IMA  personnel.  It  would  appear  that  the  mix  of 
training,  experience,  skill  levels,  etc.  embodied  in  the  man¬ 
hours  expended  in  each  maintenance  action  may  be  unique  and 
too  complex  to  provide  a  basis  for  distinguishing  among  mainte¬ 
nance  actions  on  the  basis  of  training  alone. 

Beyond  the  problems  that  can  be  laid  to  the  nature  of  ships 
themselves  (e.g.,  lack  of  standard  configurations)  and  the  basic 
features  of  this  management  system  (i.e.,  some  maintenance  is 
not  documented  and  the  remainder  is  documented  only  in  terms  of 
maintenance  actions) ,  the  form  in  which  data  are  reported  does 
not  adequately  answer  the  four  questions  basic  to  assessing 
performance  and  training  effectiveness.  The  equipment  identifi¬ 
cation  systems  (EIC  and  APL)  do  not  provide  adequate  answers  to 
what  equipment  is  maintained.  For  other  than  electronic  and 
ordnance  equipments,  an  EIC  may  reference  a  variety  of  configu¬ 
rations  with  widely  differing  maintenance  characteristics. 

While  the  EIC  system  for  electronic  and  ordnance  equipments  and 
the  APL  system  identify  precise  configurations,  they  typically 
do  so  only  at  a  system  level.  This  provides  no  way  to  isolate 
the  individual  assemblies  and  components  that,  in  fact,  fail 
and  require  maintenance,  and  the  type  and  level  of  skills  re¬ 
quired  to  repair  different  components  of  the  same  system  may 
vary  widely. 


VI.  THE  ARMY  MAINTENANCE  MANAGEMENT  SYSTEM  (TAMMS) 
AND  THE  SAMPLE  DATA  COLLECTION  PROGRAM 


Maintenance  of  Army  equipment  is  managed  through  TAMMS. 
There  is  a  noticeable  similarity  among  TAMMS,  the  Naval  Avia¬ 
tion  3-M,  and  the  Air  Force  66-1/66-5  systems  in  the  way  that 
maintenance  tasks  are  documented.  TAMMS  documents  all  mainte¬ 
nance  performed,  and  information  is  recorded  on  the  basis  of 
individual  maintenance  tasks.  Reasons  why  maintenance  was  re¬ 
quired  and  what  it  entailed  are  provided  through  code  struc¬ 
tures  that  are  quite  similar  to  the  malfunction  and  the  AT  and 
TM  codes  of  Aviation  3-M  and  66-1/66-5.  TAMMS  does,  however, 
differ  in  two  important  ways  from  both  Aviation  3-M  and  66-1/ 
66-5  as  well  as  the  Ships'  3-M  system.  The  first  is  that  its 
records  identify  neither  skill-related  WCs  nor  individuals  per¬ 
forming  maintenance.  The  second  is  that,  except  for  equipment 
modifications  and  warranty  claims,  the  maintenance  data  are 
recorded  at  the  local  level  but  are  not  transcribed  for  ADP  and 
are  not  reported  to  higher  echelons  or  a  service-wide  data 
repository.  The  only  centralized  reporting  of  comparable  data 
occurs  through  the  Sample  Data  Collection  (SDC)  Program,  dis¬ 
cussed  later,  which  is  not  a  part  of  TAMMS. 

A.  DATA  RECORDED 

Within  TAMMS,  equipment  is  identified  through  the  National 
Stock  Number  ( NSN)  system  and,  for  aircraft  and  missiles,  by 
a  Component  Breakdown  (CB)  code  that  provides  a  subsystem  iden¬ 
tification  similar  to  the  first  one  or  two  digits  of  WUC  struc¬ 
tures,  as  defined  in  MIL-STD-780E .  NSN  is  the  basic  cataloging 
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system  through  which  equipment  and  material  are  identified  by 
all  services.  NSNs  refer  to  unique  configurations  of  equipments 
(end-items,  major  systems,  subsystems,  and  assemblies)  and  to 
piece-parts  and  components  with  unique  characteristics.  While 
it  provides  a  precise  (though  possibly  cumbersome)  specification 
of  individual  material  items,  it  does  not,  by  itself,  always 
provide  an  unambiguous  identification  of  the  equipment  upon 
which  maintenance  is  performed. 

Malfunctions  are  described  by  a  set  of  Failure  Codes.  For 
aeronautical  equipment,  roughly  200  code  values  are  defined 
that,  for  the  most  part,  are  the  same  as  those  employed  by  Avia¬ 
tion  3-M  and  66-1/66-5.  For  non-aeronautical  equipment,  less 
than  30  values  are  defined,  and  these  are  dominated  by  malfunc¬ 
tions  that  are  typically  associated  with  automotive  equipment. 

What  was  done  in  the  performance  of  maintenance  of  equip¬ 
ment  is  described  by  a  set  of  Action  Codes  that  serve  the  same 
function  as  the  combination  of  codes  employed  by  Aviation  3-M 
and  66-1/66-5  (i.e.,  the  Action  Taken  and  Type  Maintenance  codes, 
the  Support  Action  Code  in  3-M  and  the  Support  General  code 
in  66-1/66-5) . 

Identification  of  who  performs  maintenance  is  limited  to 
the  naming  of  maintenance  organizations  (typically  company  units). 
Other  than  depot.  Army  Maintenance  is  structured  into  organiza¬ 
tional  and  intermediate  echelons  (with  two  intermediate  echelons 
organized  for  ground-equipment  maintenance) .  Intermediate  main¬ 
tenance  and  aircraft  and  missile  organizational  maintenance  are 
organized  into  companies  and  platoons  that  are  further  structured 
into  a  small  number  of  "sections"  that  can  be  associated  with 
broad  skill  areas  (e.g.,  electronics,  automotive).  Other  orga¬ 
nizational  echelon  units  typically  contain  a  single  maintenance 
section  that  is  responsible  for  all  equipment  belonging  to  that 
unit.  In  neither  case  does  separation  of  skill  areas  approach 
that  found  in  Naval  or  Air  Force  WCs.  It  should  also  be  noted 
that  while  all  companies  and  platoons  are  organized  into  sections 


according  to  function  performed  ( e  .g .  ,  maintenance,  supply), 
the  Army  management  system  provides  no  way  to  identify  or  dis¬ 
tinguish  among  different  sections. 

Organizational  maintenance  units  in  the  Army  are  suffi¬ 
ciently  small  so  that  a  WC  structure  does  not  appear  necessary 
or  useful  to  manage  daily  operations.  The  responsibilities  of 
maintenance  sections  may  encompass  a  range  of  skill  areas,  while 
the  small  manning  levels  frequently  permit  assignment  of  only 
one  or  two  individuals  with  appropriate  Military  Occupational 
Specialties  (MOS)  to  a  number  of  these  areas.  It  would  be  un¬ 
likely  that  the  flow  of  tasks  could  be  matched  with  available 
man-hours,  by  MOS,  in  a  way  that  would  avoid  frequent  assign¬ 
ments  of  individuals  to  tasks  outside  of  their  MOSs.  Further, 
according  to  Army  personnel,  under  combat  conditions  a  signifi¬ 
cant  fraction  of  organizational  maintenance  would  be  initiated 
by  the  dispatch  of  individuals  or  small  teams  to  the  sites  of 
disabled  equipments  on  the  basis  of  sketchy  descriptions  of  the 
nature  of  malfunctions.  The  uncertainties  regarding  what  is 
wrong  would  make  it  unlikely  that  personnel  could  be  dispatched 
on  the  basis  of  skill  area.  Under  these  conditions,  a  work 
center  structure  would  be  of  little  value  and  might  introduce 
an  element  of  inflexibility. 

B.  ASSESSMENT 

TAMMS,  as  currently  implemented,  provides  a  negligible 
capability  for  assessing  training  effectiveness.  This  negative 
assessment  rests  on  three  features  of  the  system:  the  absence 
of  central  reporting,  the  nature  of  data  that  are  recorded,  and 
the  nature  and  organization  of  maintenance  in  the  Army. 

1.  Data  Reporting 

Suprisingly,  the  absence  of  central  data  reporting  appears 
to  be  the  lesser  problem  of  the  three  noted  above.  Currently, 
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records  are  compiled  on  all  maintenance  activity  and  retained 
by  local  units  for  90  days.  All  records  older  than  this  appear 
to  be  irretrievable;  however,  central  reporting  could  be  initi¬ 
ated  at  any  time  through  a  requirement  to  submit  the  current 
hard  copy  records  to  a  central  collection  point — as  is  currently 
done  under  the  SDC  programs.  Although  expensive  and  unattrac¬ 
tive,  manual  data  extraction  and  analysis  are  possible.  Further, 
a  portion  of  currently  recorded  data  (that  pertaining  to  equip¬ 
ment  modifications  and  warranty  claims)  is  centrally  reported 
and  processed  through  ADP.  The  capability  represented  by  this 
reporting  might  provide  a  basis  for  expanding  TAMMS  into  a 
general  reporting  system. 

2.  Recorded  Data 

There  appear  to  be  some  inadequacies  in  the  data  recorded 
for  answering  the  four  basic  questions  concerning  maintenance 
performed.  While  the  NSN  system  provides  an  unambiguous  iden¬ 
tity  of  individual  equipment  and  parts,  it  is  not  wholly  satis¬ 
factory  for  assessing  performance.  First,  it  is  cumbersome  to 
use  and  quite  susceptible  to  transcription  errors;  an  NSN  con¬ 
sists  of  13  numeric  digits  of  which  only  the  first  four  denote 
the  type  of  equipment.  Second,  there  is  a  possibility  of  ambi¬ 
guities  in  the  case  of  on-equipment  maintenance;  here,  NSNs 
identify  the  major  system  or  weapon  end-item  on  which  maintenance 
is  required,  the  component  or  piece-part  at  fault,  but  not  the 
subsystem  containing  the  part.  Should  different  subsystems  of 
an  end-item  contain  the  same  component,  identity  of  the  subsys¬ 
tem  undergoing  maintenance  could  not  be  established  without 
other  information.  The  CB  code  obviates  this  problem  for  air¬ 
craft  and  missiles,  but  it  is  not  possible  to  assess  how  fre¬ 
quently  such  ambiguities  would  arise  in  the  case  of  other, 
generally  simpler,  equipment. 

Concerning  what  was  done  in  performing  maintenance,  the  use 
of  a  single  code  avoids  the  redundancy  found  in  both  Aviation 
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3-M  and  66-1/66-5  and  the  values  defined  permit  maintenance  to 
be  classified  according  to  major  function  (service,  look-phase 
of  inspections,  and  corrective).  As  with  Aviation  3-M  and  66-1/ 
66-5,  few  code  values  in  TAMMS  are  available  for  describing 
services  and  inspections.  The  values  that  are  defined  for  cor¬ 
rective  maintenance  in  TAMMS  do  not  allow  as  fine  a  distinction 
among  the  different  maintenance  tasks  that  may  comprise  a  main¬ 
tenance  action  as  do  those  of  Aviation  3-M  and  66-1/66-5.  In 
particular,  the  single  Action  Code  "repair"  appears  to  include 
on-  and  off-equipment  repair,  repair  and  replacement  of  attach¬ 
ing  and  auxiliary  parts,  assembly  and  disassembly  of  removed 
equipment,  troubleshooting,  calibration,  cleaning  and  corrosion 
repair — all  of  which  may  be  separately  identified  in  66-1/66-5. 
These  different  aspects  of  repairing  equipment  may  embody  quite 
different  levels  of  skill  and  effort  that  could  not  be  differen¬ 
tiated  by  TAMMS  data. 

3 .  Organization  of  Maintenance 

A  particular  problem  in  using  TAMMS  data  to  assess  Army 
training  effectiveness  is  that  it  is  not  possible  to  relate  par¬ 
ticular  maintenance  tasks  to  individuals  or  skill  areas  in  a 
way  that  can  be  associated  with  training  programs.  The  basic 
problem  appears  to  lie  in  Army  organization  and  operating 
environments  rather  than  in  features  of  TAMMS  data  recording. 

a.  Organizational  Maintenance.  Organizational  maintenance 
is  performed  predominantly  at  the  company  level  and  accounts 
for  roughly  one-half  of  maintenance  MOS  billets.  Companies  are 
organized  into  sections  that  can  be  associated  with  particular 
functions,  such  as  operation  of  equipment  and  maintenance  of 
equipment  (among  others),  but  there  is  no  standard  designation 
system  for  distinguishing  between  different  types  of  sections. 
Personnel  with  MOSs  in  maintenance  may  be  assigned  to  both 
equipment  operating  and  maintenance  sections.  Where  this  occurs, 
responsibility  for  maintenance  is  split  between  the  sections. 
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In  equipment-operating  sections,  minor  maintenance  and  service 
tasks  may  be  conducted  by  individuals  with  MOSs  in  equipment 
operations.  This,  along  with  the  absence  of  WCs  in  maintenance 
sections,  results  in  an  inseparable  mix  of  sections  and  training 
backgrounds  performing  maintenance  that  appears  to  preclude 
relating  maintenance  performance  to  training. 

Even  if  (1)  all  organizational  maintenance  were  performed 
by  maintenance  sections  that  were  organized  by  WC ,  (2)  the  sec¬ 
tions  and  WC  could  be  specifically  identified  in  maintenance 
documentation,  and  (3)  all  maintenance  were  performed  by  persons 
with  maintenance  MOSs,  it  would  still  be  difficult  to  relate 
performance  and  training  effectiveness.  Except  for  aircraft 
and  missile  units,  a  single  maintenance  section  is  typically 
unspecialized  and  is  responsible  for  all  equipments  held  by  a 
company  or  the  battalion  to  which  it  belongs.  Companies  and 
battalions  may  hold  a  variety  of  equipment  whose  maintenance  is 
associated  with  diverse  skill  areas  (e.g.,  wheeled  and  tracked 
automotive  equipment,  electronic  equipment,  armaments) .  How¬ 
ever,  maintenance  sections  are  relatively  small  (averaging  less 
than  20  persons),  and  it  would  appear  difficult  to  attain  either 
high  levels  of  operational  readiness  in  peacetime  or  high  equip¬ 
ment  utilization  under  combat  conditions  if  personnel  were  as¬ 
signed  to  work  only  in  their  own  specialities.  As  a  result, 
cross-skill  maintenance  should  be  a  prevalent  practice,  with  the 
implications  it  carries  for  assessing  training  effectiveness. 

b.  Intermediate  Maintenance.  intermediate  maintenance  of 
ground  equipment  is  the  responsibility  of  two  maintenance  eche¬ 
lons.  The  lower,  direct  support  (DS),  is  performed  by  units 
that  are  attached  or  organic  to  large  combat  units  (e.g.,  divi¬ 
sions)  .  The  higher  echelon  is  general  support  (GS)  and  is  per¬ 
formed  by  units  that  are  associated  with  higher  commands  such 
as  corps  and  theatre  forces.  A  single  echelon  performs  inter¬ 
mediate  maintenance  of  aircraft. 
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The  problems  faced  in  determining  training  effectiveness 
for  DS  maintenance  of  ground  equipment  and  for  intermediate 
maintenance  in  aircraft  units  are  generally  similar  to  those 
encountered  in  organizational  maintenance.  Intermediate  main¬ 
tenance  is  performed  by  battalions  and  companies  that  are  struc¬ 
tured  into  sections  associated  with  a  limited  number  of  commod¬ 
ity  areas  ( e  .g .  ,  aircraft,  components,  and  armaments  for  air¬ 
craft  units  and  automotive,  electrical,  and  engineer  equipment 
for  ground  untis) .  Organizing  sections  by  commodity  line  allows 
for  a  greater  specialization  of  both  personnel  assigned  and 
equipment  maintained.  However,  a  significant  amount  of  cross¬ 
skill  maintenance  seems  required  because  of  the  small  size  of 
sections  (typically  20  to  25),  the  limited  number  of  sections 
organized  (typically  5  to  10),  and  the  diversity  of  equipment 
maintained  by  a  DS  company.  For  example,  the  maintenance  bat¬ 
talion  of  an  armor  division  "support  command"  contains  27  hands- 
on  maintenance  sections  with  an  average  complement  of  25  person¬ 
nel  and  encompasses  eight  commodity  areas;  the  largest  electronic 
equipment  maintenance  section,  with  a  complement  of  28  personnel, 
encompasses  repair  of  surveillance  and  weapons  support  radars, 
radios,  telephone  systems,  teletypewriters,  fire  control  system 
components,  electronic  instruments,  and  special  electronic  de¬ 
vices.  Only  one  person  with  a  corresponding  MOS  is  authorized 
for  half  of  these  types  of  equipment. 

GS  maintenance  is  conducted  by  larger  organizations  whose 
sections  are  structured  into  a  larger  number  of  commodity  lines 
where  individuals  can  be  expected  to  work  only  within  their  MOS 
areas,  including  more  narrow  specialties  within  individual  MOSs . 
For  example,  under  the  "material  center"  organization,  the  GS 
maintenance  support  of  an  Army  corps  consists  of  four  centers 
with  approximately  1100  hands-on  maintenance  personnel  organized 
into  38  sections  that  encompass  23  commodity  lines  and  three 
commodity- independent  craft  areas.  However,  GS  organizations 


account  for  no  more  than  20  percent  of  Army  maintenance  authori¬ 
zations  at  combat  manning  levels;  during  peacetime  periods,  a 
significant  portion  of  assigned  personnel  are  normally  civilian 
employees.  Their  function  is  to  perform  maintenance  that  is  be¬ 
yond  the  capabilities  and  authorizations  of  lower  echelon  units 
by  virtue  of  their  more  extensive  facilities  and  the  greater 
specialization  of  personnel  that  large  size  permits.  Even  though 
GS  organizations  might  be  structured  in  a  manner  that  permits 
maintenance  performance  to  be  assessed  in  a  manner  relatable  to 
training  effectiveness,  the  types  of  tasks  performed  could  not 
be  assumed  to  be  representative  of  the  skills  employed  at  lower 
echelons  and  addressed  by  training  courses. 

c.  Operating  Environment.  Team  maintenance  also  appears 
to  be  a  complication  in  assessing  training  effectiveness  in  the 
Army.  The  size  of  a  significant  fraction  of  ground  equipment  is 
a  sufficient  reason  to  expect  extensive  team  maintenance  among 
members  of  a  maintenance  section.  In  addition,  the  environment 
of  combat  conditions  is  a  sufficient  reason  to  expect  a  differ¬ 
ent  kind  of  team  maintenance  (i.e.,  where  teams  are  comprised 
of  individuals  from  different  organizational  sections  and  eche¬ 
lons)  .  Sustaining  high  levels  of  equipment  availability  places 
a  premium  on  minimizing  the  extent  to  which  equipment  must  be 
evacuated  for  repair.  Authorizing  maintenance  personnel  to  dif¬ 
ferent  echelons  and  to  different  command  levels  and  units  of  the 
lower  echelon  permits  a  flexibility  in  dispatching  personnel  to 
effect  repairs  with  a  minimum  evacuation  of  disabled  equipment. 

The  premium  placed  on  repairs  in  the  field  during  combat 
requires  that  personnel  be  trained  in  maintaining  equipment 
under  a  variety  of  conditions  (e.g.,  location,  adequacy  of  re¬ 
pair  facilities).  This  proficiency  must  be  developed  in  peace¬ 
time  and  implies  that  maintenance  will  be  performed  under  con¬ 
ditions  ranging  from  fully  equipped  shops  to  improvised  field 
settings  with  a  minimum  of  tools.  Where  maintenance  environ¬ 
ments  differ  in  this  manner,  observed  variations  in  the  times 
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required  to  perform  otherwise  comparable  maintenance  operations 
cannot  be  taken  as  an  indication  of  differences  in  proficiency 
or  training  effectiveness. 

Similar  examples  can  be  found  throughout  the  organizational 
structure.  "Headquarters  and  Headquarters  Companies"  ( HHC)  of 
battalions  typically  contain  maintenance  sections  that  are  re¬ 
sponsible  for  the  maintenance  or  HHC  equipment  and  to  assist  the 
maintenance  and  operating  sections  of  the  battalion’s  companies. 
DS  organizations  contain  forward  support  companies,  attached  to 
brigade  headquarters,  that  are  geographically  close  to  forward 
areas  where  disabled  equipment  would  be  concentrated.  The  rear 
elements  of  DS  units  and  GS  organizations  contain  mobile  teams 
to  be  dispatched  for  forward  areas  to  assist  both  the  DS  forward 
support  companies  and  organizational  maintenance  units.  To  be 
effective  under  combat  conditions  such  arrangements  would  have 
to  be  exercised  during  non-combat  periods,  e.g.,  as  a  normal 
mode  of  operation  or  during  field  exercises  and  maneuvers,  and 
would  necessarily  be  reflected  in  maintenance  reporting. 

C.  SAMPLE  DATA  COLLECTION 

Sample  Data  Collection  (SDC)  is  specifically  directed  to 
the  evaluation  of  equipment  reliability/maintainability  and 
operation/support  costs.  It  is  implemented  as  a  series  of  indi¬ 
vidual  programs,  of  limited  durations,  for  reporting  mainte¬ 
nance  data  on  specifically  designated  equipment  using  standard 
or  modified  TAMMS  forms  or  specially  designed  forms. 

SDC  programs  address  specific  questions,  and  reporting 
forms  among  them  may  differ  to  reflect  their  diverse  interests. 
Where  modified  TAMMS  or  specifically  designed  forms  are  employed, 
it  can  be  assumed  that  they  will  entail  more  extensive  and  de¬ 
tailed  data  than  the  standard  TAMMS  format.  However,  it  cannot 
be  assumed  that  all  TAMMS  data  items  or  all  data  necessary  for 
assessing  performance  of  maintenance  personnel  would  be  relevant 
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to  and  reported  by  all  SDC  programs.  As  a  result,  generaliza¬ 
tions  cannot  be  made  regarding  whether  reported  data  satisfac¬ 
torily  identify  the  equipment  subsystems  an.’  components  that 
malfunction;  the  nature  of  malfunctions;  and  the  tasks,  man¬ 
hours,  and  individuals  or  organizations  involved  in  their  cor¬ 
rection.  Further,  data  reduction  and  evaluation  procedures 
may  have  to  be  tailored  for  each  SDC  program. 

The  volume  of  data  reported  through  SDC  is  small  relative 
to  that  recorded  through  TAMMS,  and  the  extent  of  maintenance 
effort  that  SDC  data  represent  is  similarly  small  relative  to 
total  maintenance  resources.  The  life  of  individual  SDC  pro¬ 
grams  uniformly  spans  1  year  and  can  be  extended  only  by  expli¬ 
cit  1-year  renewals.  Roughly  10  SDC  programs  are  active  at  one 
time.  The  scope  of  a  program  is  limited  to  corrective  mainte¬ 
nance  on  one  or  a  few  models  of  a  single  type  of  equipment,  and 
data  are  reported  on  a  small  fraction  of  total  inventory  of  that 
equipment.  As  examples,  during  1979  nine  SDC  programs  were  in 
effect  involving  four  models  of  artillery,  five  models  of  auto¬ 
motive  equipment,  one  missile  system,  one  command/control  sys¬ 
tem,  and  nine  model/designation/series  (MDS)  of  aircraft.  In 
the  single  program  concerned  with  the  nine  aircraft  models,  data 
collection  was  limited  to  320  aircraft  held  by  13  units  at  five 
U.S.  bases — less  than  5  percent  of  the  Army-wide  inventory  of 
the  nine  MDS.  Other  SDC  programs  are  comparably  small. 

While  these  small  samples  may  be  sufficient  to  establish 
maintenance  and  operating  cost  characteristics  of  equipment, 
they  appear  wholly  inadequate  for  assessing  performance  of  main¬ 
tenance  personnel.  However,  the  SDC  concept  may  provide  a 
foundation  for  special  data-collection  programs  for  assessing 
maintenance  personnel  performance  and  training  effectiveness. 
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VII.  SUMMARY  AND  CONCLUSIONS 


Training  effectiveness  is  currently  measured  only  by  stu¬ 
dent  achievement  at  school.  However,  the  ultimate  effectiveness 
of  training  is  determined  by  the  performance  of  personnel  on- 
the-job;  therefore,  the  comparative  effectiveness  of  different 
amounts  and  methods  of  training  should  be  measured  by  comparing 
on-the-job  performance  of  personnel  trained  in  different  ways. 

The  military  services  currently  employ  large  maintenance  manage¬ 
ment  systems  that  generate  extensive  historical  data  files.  If 
these  data  could  be  used  to  shed  light  on  the  performance  of 
either  maintenance  organizations  or  the  individuals  assigned  to 
them,  and  if  the  organizations  or  individuals  can  be  associated 
with  different  amounts  and  methods  of  training,  then  these  data 
might  also  shed  light  on  the  effectiveness  of  the  training 
alternatives . 

Five  maintenance  management  systems  were  investigated: 

The  Army  Maintenance  Management  System  (TAMMS) ,  the  Navy  Ships' 
Material  Maintenance  Management  (Ships’  3-M)  system,  the  Navy 
Aviation  Material  Maintenance  Management  (Aviation  3-M)  system, 
and  the  Air  Force  66-1  and  66-5  systems.  Taken  together,  these 
systems  encompass  organizational  and  intermediate  maintenance 
of  all  military  aircraft,  all  Army  and  Air  Force  ground  equip¬ 
ment  (including  missiles),  and  all  Navy  ships  and  shipboard 
equipment  (except  nuclear  missiles). 

We  should  not  be  surprised  to  find  that  none  of  the  systems, 
at  least  in  their  present  forms,  provides  a  suitable  vehicle  for 
assessing  training  effectiveness.  The  reasons  for  this  conclu¬ 
sion  lie  in  two  different,  but  related,  types  of  considerations. 
The  first  encompasses  rather  severe  restrictions  on  the  way 


maintenance  must  be  documented  if  maintenance  performance  is  to 
be  translatable  into  training  effectiveness.  The  second  con¬ 
cerns  ways  in  which  characteristics  of  current  military  opera¬ 
tions,  maintenance  practices,  and  equipment  may  be  inconsistent 
with  assessing  training  effectiveness.  It  should  be  noted,  how 
ever,  that  each  of  these  systems  was  designed  to  manage  mainte¬ 
nance  operations  and  were  not  meant  to  be  used  to  evaluate  the 
effectiveness  of  training  or  other  aspects  of  human  performance 

A.  DATA  RESTRICTIONS 

As  assessment  of  training  effectiveness  must  be  based  on 
comparative  analysis  limited  to  comparisons  of  performances  of 
maintenance  personnel  on  similar  maintenance  operations  (i.e., 
similar  processes  applied  to  the  same  models  of  equipment)  and 
where  personnel  are  trained  in  the  same  skill  area  by  different 
methods  or  to  different  extents.  Verifying  that  these  condi¬ 
tions  are  met  places  a  series  of  constraints  on  the  data  devel¬ 
oped  through  the  management  system,  as  follows: 

•  The  data  must  measure  the  outcome  of  maintenance  opera¬ 
tions  in  terms  that  provide  a  criterion  of  maintenance 
personnel  performance. 

•  The  data  must  provide  unambiguous  (i.e.,  coded)  answers 
to  four  questions  regarding  each  maintenance  operation: 

— What  equipment  was  maintained? 

--Why  was  maintenance  required  (i.e.,  the  nature  of 
the  equipment  malfunction)? 

--What  was  done  to  it  (i.e.,  the  nature  of  maintenanc 
performed) ? 

— Who  performed  the  maintenance? 

•  The  data  must  separately  document  discrete  and  well- 
defined  maintenance  tasks  that  are  comparable  whenever 
they  are  performed  on  the  same  subsystem  or  assembly 
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(black  box)  installed  on  the  same  model  of  equipment 
end-items . 

•  The  data  must  identify  the  equipment  maintained  at  a  suf 
ficiently  low  level  (e.g.,  subsystem  or  assembly)  so 
that  it  can  be  associated  with  a  single  skill  area  re¬ 
lated  to  specific  training  programs. 

•  The  data  must  encompass  a  sufficiently  wide  set  of  main¬ 
tenance  tasks  to  provide  a  representative  sample  of  the 
on-the-job  skill  requirements  of  a  particular  skill  area 

•  The  data  must  identify  organizations  (or  individuals) 
performing  maintenance  in  a  way  that  will  allow  their 
association  with  skill  areas  that  can  be  related  to 
specific  training  programs. 

B.  STRUCTURAL  INCONSISTENCIES 

There  are  four  characteristics  of  military  equipment,  main 
tenance  organization,  and  maintenance  practice  that,  where  they 
occur,  are  inconsistent  with  assessing  training  effectiveness 
from  on-the-job  performance  data: 

•  Maintenance  tasks  may  be  performed  by  a  group  (or  team) 
of  personnel  (i.e.,  team  maintenance). 

•  Maintenance  tasks  associated  with  one  skill  area  may  be 
performed  by  personnel  trained  in  a  different  skill  area 
(i.e.,  cross-skill  maintenance). 

•  Maintenance  organizations  may  not  be  further  structured 
into  skill-related  Work  Centers. 

•  Not  all  military  end-items  or  their  installed  subsystems 
are  built  to  standard  configurations. 

These  features  are  intractable  constraints  on  assessing  train¬ 
ing  effectiveness  from  on-the-job  performance.  However,  main¬ 
tenance  reporting  systems  might  be  designed  (or  current  systems 
modified)  to  identify  where  (on  which  maintenance  tasks)  they 


occur.  Then,  maintenance  activity  not  associated  with  these 
constraints  might  shed  light  on  training  effectiveness. 


C.  ASSESSMENT  OF  MAINTENANCE  MANAGEMENT  SYSTEMS 

Table  3  denotes,  for  each  maintenance  management  system, 
the  extent  to  which  current  system  documentation  conforms  to  the 
data  restrictions  discussed  and  where  structural  inconsistencies 
may  prevent  assessments  of  training  effectiveness.  It  was  devel¬ 
oped  through  study  of  the  systems'  documentation  and  contact 
with  knowledgeable  service  personnel.  (Note  that  a  notation  in 
Part  A  of  the  table  shows  an  imped iment  to  assessing  training 
effectiveness  while  a  notation  in  Part  B  signifies  data  that 
support  assessments.)  For  both  types  of  characteristics  the 
notations  in  the  table  give  the  maintenance  systems  (and  the 
documentation  they  generate)  the  benefit  of  the  doubt  regarding 
their  capabilities  for  assessing  training  effectiveness.  That 
is,  where  uncertainties  remained  it  was  assumed  that  the  requi¬ 
site  conditions  for  assessing  training  effectiveness  were  met. 

1.  TAMMS 

TAMMS  appears  to  provide  no  capability  for  assessing  train¬ 
ing  effectiveness.  The  Army  practices  both  team  and  cross-skill 
maintenance  in  peacetime  since  Army  units  will  operate  in  that 
fashion  under  combat  conditions.  A  major  portion  of  Army  main¬ 
tenance  units  are  not  further  structured  into  WCs  so  that  there 
is  no  way  to  identify  the  skill  areas  of  personnel  performing 
maintenance.  In  addition,  the  maintenance  reporting  format  has 
no  provision  for  noting  where  team  maintenance  occurs. 

2.  Ships'  3-M 

The  Ships'  3-M  system  appears  to  provide  no  capability  for 
assessing  maintenance  performance  (and,  hence,  training  effec¬ 
tiveness)  for  reasons  that  encompass  both  the  nonstandard  nature 


TABLE  3.  ASSESSMENT  OF  MAINTENANCE  MANAGEMENT  SYSTEMS 


Management  System 

Ships 

Aviation 

TAMMS 

3-M 

3-M 

66-1/66-5 

(Navy) 

(Navy) 

(USAF) 

Part  A: 

Structural  inconsistencies  present 

Team  Maintenance 

X 

X 

X 

X 

Cross-skill  Maintenance 

X 

X 

Maintenance  Organizations  Not 

X 

a 

a 

Structured  According  to 
Skill-Area  (i.e.  WCs) 

Non-standardized  Equipment 

X 

Part  B: 

Data  requirements  satisfied 

Quantifies  a  Criterion  of 

X 

X 

X 

X 

Performance 

Coded  Descriptions  of  Maintenance 

Performed 

•  What  Equipment  was  Maintained? 

X 

X 

X 

X 

•  What  was  wrong  with  it? 

X 

X 

X 

•  What  was  done  to  it? 

X 

X 

•  Who  Performed  the  Maintenance? 

u 

X 

X 

X 

Maintenance  "Task"  Documentation 

H 

X 

X 

Equipment  can  be  Identified  with 

1 

X 

X 

X 

Skill  Area 

1 

Recorded  Maintenance  Representa¬ 
tive  of  Job  Requirements^ 

X 

X 

Who  Performed  Maintenance  can  be 

■ 

Xc 

xd 

xd 

Identified  with  Skill-Area 

1 

aSome  WCs  are  manned  by  personnel  with  training  in  several  skill  areas. 

Recording  refers  only  to  initial  documentation  of  maintenance  performed. 

A  comprehensive  capability  for  assessing  training  effectiveness  would 
require  that  these  data  be  reported  to  higher  echelons. 

cMore  than  one  WC,  in  the  same  skill  area,  may  be  involved  in  and  iden¬ 
tified  with  the  documentation  of  one  maintenance  action. 

^Except  for  WCs  manned  by  personnel  with  training  in  different  skill  areas. 
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of  shipboard  equipment  and  the  data  reported.  Except  in  the 
area  of  electronics  and  ordnance,  shipboard  systems  are  not 
standardized,  and  ships  are  not  outfitted  to  standard  configura¬ 
tions.  Even  if  the  configuration  problem  were  not  present,  the 
data  reported  through  the  Ships'  3-M  system  appear  inconsistent 
with  assessing  training  effectiveness  on  three  counts.  First, 
maintenance  reporting  is  notably  incomplete  and  cannot  be  as¬ 
sumed  to  provide  representative  samples  of  the  ranges  of  skills 
for  which  personnel  are  trained.  Second,  data  are  reported 
only  for  complete  maintenance  actions  (as  opposed  to  individual 
maintenance  tasks).  Finally,  the  English  descriptions  and  code 
systems  used  to  document  maintenance  are  inadequate  to  identify 
comparable  maintenance  actions. 

3.  Aviation  3-M,  66-1,  and  66-5 

Even  though  the  Air  Force  systems  (66-1  and  66-5)  employ 
the  same  data-reporting  forms,  they  display  quite  different  po¬ 
tentials  for  assessing  training  effectiveness.  The  66-5  system 
appears  to  provide  no  potential  for  this  assessment.  Under  the 
POMO  structure  the  large  majority  of  maintenance  personnel  are 
assigned  to  the  organizational  echelon  where  the  development  of 
cross-skill  capabilities  is  a  primary  POMO  policy.  This  policy 
might  be  implemented  in  several  ways  (e.g.,  by  forming  inter¬ 
skill  maintenance  teams,  by  assigning  personnel  to  WCs  other 
than  those  for  which  they  have  been  trained)  .  Maintenance  data 
reporting  appears  to  provide  no  way  to  identify  cross-skill 
work  that  is  promoted  by  these  means.  In  addition,  it  is  pos¬ 
sible  that  cross-skill  maintenance  might  be  prevalent  to  the 
point  where  a  satisfactory  sample  of  in-skill  maintenance  could 
not  be  obtained. 

The  66-1  and  Aviation  3-M  systems  may  provide  a  restricted 
capability  for  assessing  training  effectiveness  if  occurences 
of  team  and  cross-skill  maintenance  could  be  unambiguously  iden¬ 
tified  and  eliminated  from  analyses.  This  might  be  possible  by 
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making  some  seemingly  modest  changes  to  the  data  reported  and 
by  supplementing  these  data  with  information  that  is  normally 
available  in  unit  roster  and  personnel  record  systems.  Whether 
the  reporting  system  changes  and  supplementary  data  suggested 
here  will,  in  fact,  provide  this  identification  requires  in¬ 
quiry  into  maintenance  operations  to  a  depth  not  accomplished 
in  this  study. 

a.  Team  Maintenance.  The  66-1  system  currently  documents 
maintenance  "crew  size"  and  start/stop  times  and  notes  all  crew 
changes  and  work  interruptions  that  occur  during  the  course  of 

a  maintenance  task.  These  data  appear  to  provide  a  satisfactory 
separation  of  team  from  individually  performed  tasks.  However, 
it  is  not  clear  whether  the  separation  provided  by  these  data 
(along  with  other  data  contained  in  maintenance  records)  are 
reliable  for  all  possible  types  of  maintenance  and  the  various 
conditions  under  which  they  may  be  performed  (e.g.,  shift  changes, 
interruptions  for  lack  of  parts,  changes  in  malfunction  diagno¬ 
sis)  .  Aviation  3-M  currently  provides  no  permanently  retained 
information  regarding  the  number  of  individuals  contributing  to 
a  maintenance  task.  However,  the  similarities  in  equipments, 
their  maintenance  requirements,  and  maintenance  organization 
between  USAF  and  Naval  aircraft  appear  sufficient  to  propose 
that  whatever  data  and  formats  would  identify  team  maintenance 
in  the  66-1  system  would  serve  the  same  purpose  in  the  Aviation 
3-M  system. 

b.  Cross-skill  Maintenance.  Both  equipments  and  wCs  ap¬ 
pear  relatable  to  skill  areas  in  the  Aviation  3-M  and  66-1  sys¬ 
tems  so  that  occurences  of  cross-skill  work  should  be  identifi¬ 
able.  However,  a  number  of  exceptions  were  noted  in  examples 
contained  in  the  Aviation  3-M  user  manuals  that  cast  doubt  on 
the  validity  of  this  conclusion.  The  extent  to  which  WCs,  in 
fact,  specialize  in  one  skill  area  (1)  may  vary  among  maintenance 


organizations  as  functions  of  size,  equipment  maintained,  and 
command  decision;  (2)  will  vary  among  the  different  WCs  within 
maintenance  organizations;  and  (3)  may  vary  over  time  within 
the  same  WC  as  a  function  of  service-wide  personnel  availabili¬ 
ties.  Further,  personnel  may  be  temporarily  transferred  between 
WCs  that  are  associated  with  quite  different  skill  areas  as  a 
result  of  workload  variations.  Similarily,  the  extent  to  which 
skill-areas  can  be  associated  with  WCs  varies.  All  such  varia¬ 
tions,  whether  they  constitute  normal  practices  or  exceptions, 
weaken  the  case  for  identifying  cross-skill  maintenance,  espe¬ 
cially  where  the  variations  are  systematic. 

c.  Reporting  System  Changes .  The  uncertain  identification 
of  both  team  and  cross-skill  maintenance  might  be  considerably 
strengthened  if  data  records  were  to  specifically  identify  all 
personnel  engaging  in  a  maintenance  task.  While  identifying 
personnel  in  the  central  maintenance  data  files  conflicts  with 
provisions  of  the  privacy  Act,  it  should  be  possible  to  name 
individuals  in  local  unit  records  and  then  to  process  these 
records  in  ways  that  would  provide  suitable  identification  for 
analyses  of  training  effectiveness  while  being  consistent  with 
the  Act.  For  example,  individuals  could  be  preselected  for 
analysis  of  performance  on  the  basis  of  training  and  experience 
information  in  personnel  records.  Look-up  tables  could  be 
established  or  special  notations  (flags)  could  be  attached  to 
their  names  or  identification  numbers  as  a  device  for  identify¬ 
ing  the  maintenance  tasks  they  subsequently  perform.  Where 
maintenance  documentation  identified  these  individuals  and 
where  neither  team  nor  cross-skill  maintenance  was  indicated, 
the  records  would  be  duplicated  for  analysis  of  performance, 
and  the  identification  of  individuals  could  then  be  deleted 
from  the  maintenance  records  submitted  to  central  files. 
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Analyses  of  training  effectiveness  could  then  be  performed 
without  interfering  with  either  the  current  maintenance  manage¬ 
ment  systems  or  the  organization  of  maintenance.  ADP  of  the 
Aviation  3-M  system  is  currently  being  modified  to  accomplish 
on-line  record  entry  and  updating  (instead  of  hardcopy  and 
punched  card)  as  part  of  the  Navy  Air  Logistics  Command  Manage¬ 
ment  information  System  (NALCOMIS)  program.  As  presently  de¬ 
signed,  local  unit  on-line  records  will  identify  personnel  by 
name,  and  this  information  will  be  dropped  when  the  local  rec¬ 
ords  are  committed  to  the  Aviation  3-M  central  tape  files.  It 
appears  to  be  a  feasible  further  step  to  provide  identification 
in  a  way  that  would  allow  selection  of  the  on-line  records  for 
performance  analysis.  An  added  benefit  of  this  approach  would 
be  that  analyses  could  proceed  more  directly--without  the  inter¬ 
mediate  step  of  modelling  and  regression  analyses. 

With  these  changes,  the  nagging  question  remains  of  whether 
the  resulting  samples  would  be  representative  of  the  job  require¬ 
ments  of  skill  areas.  The  prospects  are  not  promising,  espe¬ 
cially  at  the  organizational  echelon.  An  analysis  that  concen¬ 
trated  on  recently  trained  personnel  would  be  essentially  limited 
to  personnel  who  are  not  qualified  for  independent  work  on  air¬ 
craft  equipment.  The  Air  Force  has  a  service-wide  procedure 
for  job  qualification  that  consists  of  structured  OJT  and  per¬ 
formance  examinations,  and  personnel  are  not  permitted  to  per¬ 
form  independent  work  until  specifically  qualified.  (The  Navy 
also  has  defined  qualification  standards  and  OJT  programs,  al¬ 
though  the  Navy  program  is  less  restrictive  regarding  the  work 
which  can  be  accomplished  by  new  personnel.)  It  could  well  be 
that  by  the  time  an  individual  meets  the  qualification  standards 
for  independent  work  the  impact  of  different  formal  training 
programs  would  be  diluted  to  the  point  where  initial  differences 
in  performance  that  were  due  to  the  training  would  be  washed 
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GLOSSARY 

Cross-Skill  Maintenance:  Performance  of  maintenance  that  is 
associated  with  one  skill  area  by  personnel  that  are 
trained  in  a  different  skill  area. 

Maintenance  Action:  All  effort  associated  with  a  maintenance 
requirement  that  permits  equipment  to  be  returned  to 
operational  status. 

Maintenance  Task:  Individual  well  defined  maintenance  proce¬ 
dures  that  are  parts  of  maintenance  actions. 

Off-equipment  Maintenance:  Maintenance  performed  on  equipment 
systems  and  assemblies  that  have  been  removed  from  weapon 
end-itmes. 

On-equipment  Maintenance:  Maintenance  performed  on  equipment 
systems  and  assemblies  while  they  are  installed  on  weapon 
end- items . 

Team  Maintenance:  Separately  documented  maintenance  actions  or 
tasks  that  are  performed  by  more  than  one  person. 

End-item:  The  combination  of  systems,  components,  and/or  mate¬ 
rial  that  is  complete  for  its  intended  use,  e.g.,  ship, 
tank,  aircraft,  mobile  machine  shop. 
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ACRONYMS 


» 

AAC 

A  DP 

AMES 

AN 

APL 

AT 

Alaskan  Air  Command 

Automatic  Data  Processing 

Aircraft  Maintenance  Effectiveness  Simulation 

Army  Navy 

Authorization  parts  List 

Action  Taken 

CB 

Component  Breakdown 

t 

DCNO 

DCS 

DS 

Deputy  Chief  of  Naval  Operations 

Deputy  Chief  of  Staff 

Direct  Support 

EIC 

Equipment  Identification  Code 

r 

GS 

General  Support 

HHC 

Headquarters/Headquarters  Company 

IMA 

Intermediate  Maintenance  Activity 

* 

JCN 

job  Control  Number 

» 

MAC 

MDS 

Mk/Mod 

Mil  Std 

MOS 

Military  Airlift  Command 

Model/Designation/Series 

Mark/Model 

Military  Standard 

Military  Occupational  Specialty 

t 

NALCOMIS 

NAMP 

NAVAIR 

NIP 

NORM 

NSN 

Navy  Air  Logistics  Command  Management  information 
System 

Naval  Aviation  Maintenance  Program 

Naval  Air  Systems  Command 

NAMP  Improvement  Program 

Not  Operationally  Ready  -  Maintenance 

National  Stock  Number 

OJT 

On-The-job  Training 

9 

PACAF 

POMO 

pacific  Air  Forces 

production  Oriented  Maintenance  Organization 

I 
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SAC 

SDC 

SEL 


Support  Action  Code 
Sample  Data  Collection 
Selected  Equipment  List 


TAC 

TAMMS 

TM 

USAFE 

WC 

WUC 

3-M 

66-1 


66-5 


Tactical  Air  Command 

The  Army  Maintenance  Management  System 
Type  Maintenance 

United  State  Air  Forces  in  Europe 

Work  Center 
Work  Unit  Code 

Maintenance  and  Material  Management 

The  Air  Force  maintenance  management  system  employed 
by  all  Air  Force  organizations  other  than  units  of 
the  Tactical  Air  Forces.  The  name  is  derived  from 
the  number  of  the  Air  Force  regulation  and  manual 
that  sets  out  maintenance  policy  under  this  system. 

The  Air  Force  maintenance  management  system  employed 
by  units  of  the  Tactical  Air  Forces  (AAC,  PACAF,  TAC, 
USAFE)  that  are  organized  according  to  the  POMO  con¬ 
cept.  The  name  is  derived  from  the  number  of  the 
Air  Force  regulation  and  manual  that  sets  out  main¬ 
tenance  policy  under  this  system. 
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APPENDIX 


DATA  RECORDING  FORMS 

This  appendix  discusses  the  forms  on  which  maintenance  data 
are  recorded  in  each  of  the  management  systems.  The  forms  re¬ 
flect  the  major  functions  of  the  systems  (daily  management  of 
maintenance  operations,  configuration  control,  and  material  sup¬ 
port  and  control),  and  a  limited  fraction  of  the  recorded  infor¬ 
mation  bears  directly  on  answering  the  four  basic  questions  rele 
vant  for  assessing  maintenance  performance  and  training  effec¬ 
tiveness.  In  the  displays  of  the  data  recording  forms,  those 
fields  that  are  considered  relevant  for  assessing  training  effec 
tiveness  are  outlined  and  numbered,  and  the  numbers  serve  as 
references  in  the  subsequent  discussions.  For  the  three  systems 
that  employ  EDP  (Ship's  3-M,  Aviation  3-M,  and  66-1/66-5)  the 
data  fields  that  neither  are  transcribed  to  EDP  records  nor  are 
relevant  for  assessing  training  effectiveness  have  been  shaded. 

A . 1  NAVAL  AVIATION  3-M 

The  VIDS/MAF  form,  Figure  A-l,  is  used  for  recording  infor¬ 
mation  on  all  corrective  maintenance  and  the  look-phase  of  major 
inspections.  All  information  transfered  to  EDP  consists  of  code 
values  (either  codes  developed  specifically  for  recording  main¬ 
tenance  information  or  codes  that  serve  other  purposes  such  as 
part  and  manufacturer  numbers)  and  numeric  values  such  as  dates 
and  quantities. 

The  equipment  maintained  is  identified  by  the  WUC  value 
for  the  specified  type  of  equipment  (end-item),  (2).  The  reason 
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REMOVED/OLD  ITEM 
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JNSTAILED/NEW  ITEM 

GO*  MFC*  Gl  3  SCNIAl  NUM*EN 


Figure  A-l .  VIDS/MAF  Form 


maintenance  is  performed  is  given  by  the  malfunction  code,  (5). 
What  was  done  in  the  course  of  the  maintenance  is  described  by 
the  action  taken  (4)  and  type  maintenance  (7)  codes.  Who  per¬ 
forms  maintenance  is  identified  by  the  combination  of  work 
center  (?)  and  the  maintenance  organization  to  which  it  belongs 


Total  man-hours  expended  on  the  task  are  recorded  in 


The  individuals  performing  maintenance,  along  with  the  man-hours 


expended  by  each,  are  separately  identified  in  the  "accumulated 
work  hours"  section  (T)  ;  however,  this  information  is  not  trans- 


fered  to  EDP  records. 


A. 2  AIR  FORCE  66-1  AND  66-5 


The  Maintenance  Data  Collection  Record  (Figure  A-2)  is  used 
to  record  all  maintenance  in  the  66-1/66-5  systems.  Equipment 
being  maintained  is  identified  by  the  values  of  the  WUC  4  and 
the  SRD  (2).  ( SRD ,  or  Standard  Reporting  Designator,  is  a  three 

character  code  that  denotes  the  type  of  equipment  end-item.) 

What  is  wrong  with  the  equipment  is  given  by  the  how-malf unction 
code  (?)  .  What  was  done  in  the  course  of  maintenance  is  con¬ 
tained  in  the  type-maintenance  (?)  and  action-taken  (?)  codes. 

Who  performed  the  maintenance  task  is  identified  through  the 
combination  of  work  center  (?)  and  the  maintenance  organization 
containing  the  work  center,  where  the  identification  of  the 
organization  is  provided  by  the  processing  routines  that  trans¬ 
fer  the  hard-copy  information  to  EDP  records.  Man-hours  expended 
are  calculated  during  initial  EDP  from  the  start-hour,  stop-hour, 
and  crew  size  values  contained  in  (?)  .  A  separate  line  of  in¬ 
information  is  recorded  each  time  there  is  a  change  in  any  of 
the  columns  A  through  N,  including  changes  in  crew  size  or  com¬ 
position,  and  each  line  results  in  a  distinct  EDP  record.  Em¬ 
ployee  number  (?)  typically  names  a  shop  supervisor  or  lead 
technician  and  should  not  be  a  reliable  guide  to  identifying 
individuals  actually  performing  maintenance;  in  addition,  this 
information  is  not  transfered  to  EDP  records. 


Figure  A-2.  Maintenance  Data  Collection  Record 
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Figure  A-2  (Continued).  Back  of  Maintenance 
Data  Collection  Record 


A. 3  NAVY  SHIP'S  3-M 


Several  reporting  forms  are  employed  in  the  Ship's  3-M 
system,  and  the  same  information  may  be  displayed  on  different 
forms.  Two  of  these  forms  are  completed  manually  by  ships  force 
and/or  IMA  personnel  and  serve  to  initiate  EDP  records  of  main¬ 
tenance  actions.  The  EDP  records  may  be  updated  and  used  to 
generate  work  status  reports  before  the  maintenance  is  completed 
and  as  a  final  report  upon  completion. 

All  information  pertaining  to  maintenance  actions  that  are 
completed  by  ship's  force  personnel  is  manually  recorded  on  the 
Ship's  Maintenance  Action  Form  (2-KILO),  Figure  A-3,  and  all 
but  a  few  items  are  transcribed  to  EDP  records.  The  equipment 
being  maintained  is  identified  by  either  or  both  the  APL  or  EIC, 
(3)  and  (4).  The  remarks  description  section,  (?)  allows  for 
English  descriptions  of  information  not  provided  for  in  other 
sections  of  the  form;  this  is  the  only  place  for  describing  what 
was  wrong  with  the  equipment  and  what  was  done  to  effect  repair. 
(The  EDP  records  permit  descriptions  up  to  1200  characters.) 
Identification  of  who  performed  the  maintenance  is  accomplished 
through  the  combination  of  the  ship's  identification  code  number 
(UIC)  and  the  performing  WC,  (T)  and  (2).  Where  maintenance  is 
deferred,  the  ship's  force  man-hours  expended  to  the  date  of 
deferral  are  recorded  at  (5)  and  an  estimate  of  the  man-hours 
required  for  completion  is  recorded  at  (?) .  When  a  deferred 
action  is  completed,  the  total  ship's  force  man-hours  expended 
are  recorded  at  (7)  • 

When  a  maintenance  action  is  completed  with  IMA  assistance 
the  second  manual  form  is  completed  by  IMA  personnel  for  plan¬ 
ning/scheduling  of  the  IMA  effort.  The  information  contained 
on  the  Ship's  Maintenance  Action  Form  and  the  planning/schedul¬ 
ing  information  are  consolidated  into  a  single  EDP  record  (the 


A- 6 


3-M  Master  Record) .  This  record  is  used  to  generate  the  Auto¬ 
mated  Work  Request,  Figure  A-4,  and  contains  all  but  a  few  of 
the  data  items  contained  in  the  3-M  Master  Record. 

The  3-M  Master  Record  contains  all  of  the  items  in  the 
Ship's  Maintenance  Action  form  (2-KILO)  that  are  considered  as 
possibly  useful  for  assessing  training  effectiveness,  and  the 
Automated  Work  Request  displays  all  but  one  of  these  items,  the 
ship's  force  man-hours  expended  through  the  date  of  deferral 
(?)  .  In  addition  it  displays  information  regarding  the  IMA 
assistance  provided,  including  identification  of  the  IMA  (12) 
and  its  various  WCs  providing  the  assistance  (?) ,  the  man-hours 
expended  by  each  of  the  IMA  WCsT'lo')  ,  and  a  cryptic  description 
of  the  task  performed  by  each  (llj".  At  the  completion  of  the 
maintenance  action  the  3-M  Master  Record  (from  which  the  Auto¬ 
mated  Work  Request  is  generated)  documents  the  total  effort 
expended  by  both  the  IMA  and  the  tended  ship. 


A. 4  ARMY  TAMMS 


In  TAMMS  all  maintenance  data  are  recorded  on  the  Mainte¬ 
nance  Request,  DA  Form  2407,  Figure  A- 5.  Reasons  for  performing 
maintenance  are  given  by  the  Failure  Code  (?) ;  what  was  done  to 
effect  repairs  is  given  by  the  Action  Code  (?) ;  the  man-hours 
expended  are  shown  in  (?)  .  Each  change  in  the  action  or  failure 
codes  (or  the  component,  assembly,  or  piece  part  of  equipment 
involved  in  the  maitenance)  is  documented  on  a  separate  line 
of  these  fields.  This  provides  for  separate  documentation  of 
the  individual  maintenance  tasks  while  one  Maintenance  Request 
form  can  encompass  a  complete  maintenance  action. 

Identification  of  who  performs  maintenance  is  accomplished 
by  the  Unit  Identification  Code  assigned  to  all  Army  units. 

Where  maintenance  is  accomplished  at  the  intermediate  echelon 
( DS  or  GS)  the  code  of  the  performing  unit  is  entered  in  (4). 

For  organizational  echelon  maintenance,  the  code  of  the  performing 


Figure  A-5.  Maintenance  Request,  DA  Form  2407 
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unit  is  entered  in  (T)  and  (4)  and  will  be  blank..  However, 
with  the  current  Army  management  structure  the  lowest  echelon 
unit  assigned  a  Unit  Identification  Code  will  likely  be  the 
equipment  custodian,  responsible  for  both  equipment  operations 
and  all  maintenance  (e.g.,  a  tank  company  containing  both  oper¬ 
ating  and  maintenance  sections  or  a  tank  battalion  that  contains 
a  maintenance  platoon  to  provide  support  to  both  its  own  equip¬ 
ment  and  the  equipment  of  the  tank  companies  it  contains) . 

(Note  that  block  7  of  the  form,  maintenance  activity,  is  limited 
to  identifying  the  echelon  at  which  the  maintenance  is  accom¬ 
plished.  ) 

The  way  in  which  equipment  being  maintained  is  identified 
will  differ,  depending  upon  whether  on-equipment  or  off-equip¬ 
ment  maintenance  is  performed  and,  for  on-equipment  work,  whether 
aircraft  or  ground  equipment  is  involved.  As  a  result,  the 
labelling  of  the  columns  and  boxes,  especially  block  20  of  the 
form,  may  not  reflect  the  data  they  actually  contain.  The  data 
contained  in  relevant  sections  of  the  form  are  summarized  in 
Table  A-l  and  discussed  below. 

Table  A-l.  IDENTIFICATION  OF  EQUIPMENT  MAINTAINED 


End- Item 

Subsystem  (of  End-Item) 

System  Removed  (From  End-Item) 

Component/Assembly  of 
System  Removed 

Assembly  or  Parts  Repaired 
and/or  Replaced 


Off-Equipment 

On-Equipment 

:  Maintenance 

Ma i ntena nee 

Aircraft  Ground  Equipment 

- 

© 

© 

- 

0 

- 

© 

- 

- 

© 

- 

- 

© 

© 

©£© 

®*  © 

For  of  E-equipment  maintenance,  the  NSN  of  the  system  re¬ 
moved  from  an  end-item  (e.g.,  a  communication  set)  is  inserted 
at  (T)  ,  and  the  model  name  and  NSN  of  the  component  or  assembly 
of  the  system  that  is  actually  undergoing  maintenance  (e.g., 
the  transmitter  section  of  the  communication  set)  is  inserted 
at  (2) .  For  individual  maintenance  tasks  that  pertain  to  the 
assembly  as  a  whole  (e.g.,  adjustment  or  inspection)  (s)  and 
(To)  will  also  show  the  model  name  and  NSN  of  the  assembly;  for 
tasks  involving  replacement  of  parts  within  the  assembly,  the 
names  of  the  parts  and  their  NSNs  will  be  inserted  in  (s)  and 

©• 

For  on-equipment  maintenance  of  aircraft,  the  model  number 
and  NSN  of  the  aircraft  (i.e.,  the  end-item)  will  be  contained 
by  (2)  ,  and  identification  of  the  subsystem  requiring  mainte¬ 
nance  will  be  provided  by  the  CB  code  (similar  to  the  first  two 
digits  of  Navy  and  Air  Force  WUCs)  at  (7)  .  (s)  will  contain  a 

description  of  individual  maintenance  tasks  performed  or  parts 


replaced,  and  where  parts  are  used,  yKy  will  contain  the  NSN 
of  those  parts.  For  ground  equipment  (2)  will  contain  the 
model  number  and  NSN  of  the  end-item.  (b)  will  contain  the 
name  of  the  system,  assembly  or  part  undergoing  repair  or  being 
replaced.  When  an  assembly  or  part  is  replaced  (lo)  will  con¬ 


tain  its  NSN. 
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